12 Ways to Improve Public Confidence in Gender Pay Gap Data

This is intended to be a briefing note for anyone interested in seeing gender pay gap data being used properly. Gender pay gap data is now part is the business and political discourse in the UK and is likely to be so for some time. If the goal of gender pay gap reporting is to remove disparities in pay between men and women, then it is essential that the public has confidence in both the quality of the data being published and the correctness of any interpretations of the results. With the first round of data reporting out of the way, it is time to learn lessons, identify improvements and see that these are implemented. Following extensive analysis of the 2017 round of results, I have identified 12 ways to improve the quality of the data being reported and make interpretations of the results more meaningful.

Background to this briefing note

I have written a number of articles about the gender pay gap covering these topics:-

It is worth reading these so that you are familiar with the terms and ideas I will be discussing in this article (such as the difference between equal pay and gender pay gap) but I will refer to these articles again where relevant.

How can gender pay gap data be improved?

My 12 suggestions cover two themes, ensuring the reported data is accurate and making interpretations of the data more meaningful. Some suggestions will cover both and I will make this clear when I expand on each suggestion. For now, the 12 suggestions below are coloured in red if it is intended to improve data quality, in blue if it intended to make interpretations more meaningful and in black if it is intended to do both.

Organisations should report the median woman’s earnings/bonus in pounds and pence, assuming that the median man earnings/bonus is £1.00.
Ask someone skilled in explaining basic statistics to non-statistical audiences to rewrite the guidance on how the calculations are done.
Put a sanity check on the data entry website to prevent impossible values being entered.
Include a sanity check to compare the median gender pay gap with the female income quartile gap.
For each income quartile the median gender pay gap should be reported in same format as suggestion 1.
Abolish the requirement to report the mean (or average) earnings and bonuses.
Offer free tools (spreadsheet or otherwise) to enable organisations to do their calculations correctly and upload their results without error.
Organisations who employ less than 100 women should be flagged.
Once an organisation uploads data for two years running, publish both years’ results side by side on the government website.
Allow an organisation’s gender pay gap report to be rated thus identifying good and bad examples.
Do not lower the minimum threshold of 250 employees.
Use a different process to measure racial & disability pay gaps i.e. do not replicate the gender pay gap reporting process.

For each suggestion listed, I will explain my thinking first and then finish with a paragraph headed Recommendation. This will describe the actions that I would like to see undertaken.

1 – Organisations should report the median woman’s earning/bonus in pounds and pence.

The government’s website reports the median gender pay gap in two different ways. Below I have copied and pasted the reported figures from the links shown for two different organisations which are presented in two different ways. Which way of reporting do you find most intuitive and helpful?

First the Financial Conduct Authority.

Women’s median hourly rate is 20.9% lower than men’s
In other words when comparing median hourly rates, women earn 79p for every £1 that men earn.

Second, Nestle Purina UK Ltd.

Women’s median hourly rate is 21% higher than men’s
In other words when comparing median hourly rates, women earn £1.21 for every £1 that men earn.

Personally, I find the second format of pounds and pence more intuitive. If an organisation says that their pay gap is 21%, they need to provide extra information as to whether that is in favour of men or women. If you adopt the convention of defining the median man as earning £1, then it is immediately obvious that an organisation that pays the median woman £0.79 is favouring men overall and an organisation paying the median woman £1.21 is favouring women overall.

However, problems are arising because the government is asking organisations to submit their median gender pay gap in the first format. If you read the government’s instructions for calculating the median gender pay gap, you will see that organisations are being asked to use this formula.

100% x ( Median Male Hourly Earnings – Median Women’s Hourly Earnings ) / Median Male Hourly Earnings

The problem is that there are in fact four ways the median gender pay gap can be calculated as a percentage as shown below and none of the four options are superior to the others. It is a matter of choice of which to use and the government has chosen to use the first option:

100% x ( Median Male Hourly Earnings – Median Female Hourly Earnings ) / Median Male Hourly Earnings
100% x ( Median Female Hourly Earnings – Median Male Hourly Earnings ) / Median Male Hourly Earnings
100% x ( Median Male Hourly Earnings – Median Female Hourly Earnings ) / Median Female Hourly Earnings
100% x ( Median Female Hourly Earnings – Median Male Hourly Earnings ) / Median Female Hourly Earnings

What is undoubtedly happening is that some organisations are getting confused and using options 2, 3 or 4 instead. If the wrong option is chosen, the results can be very different. Suppose in an organisation, the median man earns £25 per hour and the median woman earns £20 per hour. Then these four options will give the following median gender pay gaps when expressed as a percentage.

+20% = 100% x (25-20) / 25
-20% = 100% x (20-25) / 25
+25% = 100% x (25-20) / 20
-25% = 100% x (20-25) / 20

On the other hand, if organisations are asked to submit the hourly earnings of the median woman, assuming the median man earns £1, then there is much less room for confusion. Let’s take our example of where the median man earns £25 per hour and the median woman earns £20 per hour. Having determined these two numbers, all the organisation has to do is divide both numbers by the median man’s figure of £25ph. This will result in £1 for the median man and £0.80 for the median woman and the latter figure is all that needs to be reported.

One final point, you will notice that I have constantly referred to the median man and the median woman i.e. I have tried to convey the image of individual people. I would like to see this language adopted throughout rather than the more abstract form such as “women’s median earnings”.

RECOMMENDATION

The website where organisations have to submit their figures should be changed. Organisations should be asked to report the median woman’s hourly earnings in pounds and pence, assuming that the median man’s hourly earnings is £1.

PS: As this article was being written, the Government Equalities Office came to the same conclusion. They commissioned research to find the best way to present pay gap data to the public and they concluded I was right! At the moment though, all that has changed is the way the data is presented on the government website, not the website where organisations have to submit their data.

2 – Ask someone skilled in explaining basic statistics to non-statistical audiences to rewrite the guidance on how the calculations are done.

I have identified countless examples of organisations who have clearly entered incorrect data. This prompted me to review the government’s guidance on the calculations that had to be done and I am afraid that what I saw was open to misinterpretation. A good example is the instructions for calculating the median gender pay gap which I reproduce below:

Median gender pay gap in hourly pay: how to calculate

Arrange the hourly pay rates of all male full-pay relevant employees from highest to lowest
Find the hourly pay rate that is in the middle of the range – this gives you the median hourly rate of pay for men
Arrange the hourly pay rates of all female full-pay relevant employees from highest to lowest
Find the hourly pay rate that is in the middle of the range – this gives you the median hourly rate of pay for women
Subtract the median hourly pay rate for women from the median hourly pay rate for men
Divide the result by the median hourly pay rate for men
Multiply the result by 100 – this gives you the median gender pay gap in hourly pay as a percentage of mens’ pay

Steps 2 and 4 are an extremely common description of how to calculate the median but I know from my own experience of training people in basic statistics that many people interpret this to mean the middle point between the minimum and maximum values. Take the graphic shown here where we have 7 men and 5 women earning between £10 & £50 per hour. So far steps 1 and 3 have been complied with since each gender is lined up in order of their hourly earnings. Now look closely at the words of steps 2 & 4 where it says “middle of the range”. Interpreted literally, this can be read as the middle of the range £10 to £50 i.e. £30 per hour. What steps 2 & 4 should be saying is “middle of the line” i.e. the 3rd woman in the line of 5 women and the 4th man in the line of 7 men, both of which are highlighted in the graphic.

Later on, instructions are given on how to calculate the quartiles and the same semantic error is repeated. When instructed to “divide this into 4 equal parts”, some organisations are clearly interpreting this to mean the range £10 to £50 needs to be split into 4 equal sized ranges i.e. £10-20, £20-30, £30-40 and £40-50 per hour. Eastleigh Borough Council is a good example where their published income quartiles contain differing numbers of employees in each quartile. Again the correct instruction is that the line of all employees (not the range) needs to be split into 4 parts with equal numbers of employees in each part. You can see that I have done this with our 12 employees in the graphic below.

The guidance as it stands is open to misinterpretation and could be a lot better. Some graphics like the ones I have used would be helpful but most of all, it needs to be rewritten by someone who is skilled in explaining basic statistical concepts. The vast majority of HR managers and directors will not have strong stats skills and the guidance has to be written and tested with this audience in mind.

RECOMMENDATION

The guidance about doing the calculations should be rewritten by someone skilled in explaining basic statistical concepts to a non-statistical audience. The rewritten guidance should also be tested on a suitable sample of HR managers and directors. I am sure that a trade body representing such people would be happy to cooperate with the testing.

3 – Put a sanity check on the data entry website to prevent impossible values being entered.

The confusion about how to calculate the median gender pay gap that I describe in suggestion #1 could explain why some impossible values are being entered by some organisations. For example, Shrewsbury Academies Trust say their gender pay gap is +121%. But if you work back from the 1st formula listed in suggestion #1, you will find that this equates to the median woman earning -£0.21 as the link shows. Clearly the website the organisations are using to enter their data does not have a sanity check to prevent such impossible values.

This where adopting my suggestion #1 would help, namely that organisations should be entering the median woman’s hourly earnings assuming the median man’s hourly earnings is £1. I am sure if Shrewsbury had tried to enter -£0.21 they would immediately realise something can’t be right. It might also prompt organisations that have large pay gaps favouring women to check their figures. For example, Randstad HR claim they pay their median woman £2.03 which surely would prompt a second look. When you click on their own report, you find that they are actually saying the median woman earns £0.91.

RECOMMENDATION

The website where organisations enter their data should have sanity checks installed. Clearly situations where the median woman earns less than £0.00 must be rejected but I would also have an automatic query along the lines of “are you sure?” where the median woman earns less than £0.50 or greater than £2.00.

4 – Include a sanity check to compare the median gender pay gap with the female income quartile gap.

The sanity check in suggestion #3 is intended to capture silly numbers being entered and will only affect a few organisations. A far more common error occurs when organisations enter their gender splits by quartile. What is not appreciated is that these figures can provide a sanity check on the median gender pay gap. To do this, organisations need to calculate their Female Income Quartile Gap (FIQP). I will use DFDS Seaways as an example where the median woman earns £0.96 as shown in the chart.

The Female Income Quartile Gap (FIQP) is calculated by:

Noting the % that are female in the Upper Income Quartile, 39% in this case
Adding the % that are female in the Upper Middle Income Quartile, 25% in this case
Subtracting the % that are female in the Lower Middle Income Quartile, 19% in this case
Subtracting the % that are female in the Lower Income Quartile, 24% in this case

For DFDS Seaways, this works out to be +21% = 39% + 25% – 19% – 24%. It is not difficult to show that when the FIQP is positive, the median woman should be earning more than £1.00 and when the FIQP is negative, the median woman should be earning less than £1.00. Recall the graphic I used in suggestion #2 to demonstrate that the median man and woman stands in the middle of their lines and that each income quartile should have equal number of employees. Now let’s suppose that DFDS employ exactly 400 people which would mean each income quartile consists of 100 people. According to the chart, if these 400 people separated themselves by gender and sorted themselves by hourly earnings, the female line would have 107 women (treating the %’s in the chart as actual numbers instead) and the male line would have 293 men. Therefore the median woman would be the 54th woman in the line and the median man the 147th man in the line.

Which income quartiles do the 54th woman and the 147th man lie in? From the chart and starting from the lower income quartile, we find that the 54th woman must lie in the upper middle income quartile since the chart says there are 43 women in the bottom two quartiles and 25 woman in the upper middle income quartile. By contrast, the 147th man lies in the lower middle income quartile since we have 157 men in total in the bottom two quartiles. Therefore the median woman earns more than the median man but DFDS Seaways says the median woman earns £0.96 so there is a conflict here.

This rule is not cast-iron but I have shown through simulations that it should hold 99% of the time. What is certain is that if an organisation submits a median gender pay gap that conflicts with the Female Income Quartile Gap, then a warning should appear asking them to check their data. From what I have seen, the most common reason is that the gender splits for the income quartiles have been entered in the wrong order i.e. DFDS Seaways say 39% of the Upper Income Quartile are women but in fact this figure should have been entered in the Lower Income Quartile instead.

Another situation where this sanity check works is when an organisation says the median woman earns £1.00 i.e. there is no gender pay gap but the FIQP is clearly positive or negative. Again this would be a conflict that needs to be investigated. My belief is that the organisation has failed to understand the government’s guidance on calculating the median man and median woman (as explained in suggestion #2) and they have taken the “middle of the salary ranges” to be the median rather than the middle of the lines. For some reason, football clubs are prone to this error and three are shown in this graphic. In all cases, the FIQP points to the median women earning less than £1.00.

RECOMMENDATION

The government should install the Female Income Quartile Gap sanity check on the website where organisations enter their data and use this to challenge organisations that enter conflicting data.

5 – For each income quartile, the median gender pay gap should be reported in same format as suggestion 1.

If an organisation has gone to the trouble of splitting their employees into income quartiles, then it should be straightforward to for them to calculate the median gender pay gap within each quartile. The graphic here demonstrates what I am proposing.

This company has 24 employees, 12 women and 12 men, each standing in order of their hourly earnings. When you have an even number of people, the median is found by taking the average of the two people in the middle of the line. For both men and women, these are the 6th (£18ph) and 7th (£22ph) persons in the line and so both the median man and median woman are both earning £20 per hour, hence no gender pay gap.

It’s get better as far as the official statistics go. Within each income quartile, there is a 50:50 male:female split which means the company is 50:50 overall. Combined with the median gender pay gap of zero, this company sounds ideal! Yet by now you should have spotted that within each income quartile, the 3 men are being paid more than the 3 women and therefore this company probably is breaking the law on equal pay.

Such a situation could easily arise if organisations decide to deal with their pay gaps by gaming the system. Whenever people are assessed against performance targets, gaming is something that nearly always occurs. In this situation, all the employer has to do is ensure that the median man and median woman are paid the same. Above and below those two people, it does not matter what happens.

To pick up situations like this, I propose that for each income quartile, the median gender pay gap is calculated. You can see my calculations at the bottom of the graphic and the median man and median woman in each quartile is highlighted. These figures use the format recommended in suggestion #1 and clearly show gender pay gaps. Whilst this system is not perfect, it will make it more difficult for an unscrupulous organisation to game the system.

RECOMMENDATION

Within each income quartile, organisations should have to report the median gender pay gap using the format of suggestion #1 i.e. reporting the median woman’s earnings in pounds and pence assuming the median man earns £1.

6 – Abolish the requirement to report the mean (or average) earnings and bonuses.

You might argue that the situation in the graphic in suggestion #5 could be picked up if you looked at the mean (or average) gender pay gap. In that example, the average man earns £22.50 and the average woman earns £20.17 which means for every £1 the average man earns, the average woman earns £0.90. So why not keep the mean gender pay gap reporting requirement?

There are two reasons why I recommend the abolition of the requirement to report the mean gender pay gap for bonuses and earnings. The main one is that is that it only takes one multi-millionaire to distort the mean gender pay gap. The graphic below is the same company used in suggestion #5 i.e. 12 men and 12 women, and this time, the median gender pay gap is zero both overall and within each quartile. But the mean gender pay gap (average man is paid £30ph, average woman £22.50ph) is significant due solely to the highest paid man being paid considerably more than the highest paid woman. Otherwise, the impression of the company overall looks good but you would not know this from the mean gender pay gap.

The second reason I would like to see the mean gender pay gap abolished is that at the moment, promoters and detractors of an organisation can whichever of the two gender pay gaps best support their case. This is a clear misuse of gender pay gap data and will only fuel distrust. By abolishing the mean gender pay gap and leaving just the median gender pay gap, there will be only one figure to play with.

RECOMMENDATION

The government should abolish the requirement to report the mean gender pay gap for hourly earnings and bonuses.

7 – Offer free tools to enable organisations to do their calculations and upload their results without error.

In suggestion #1 and #2, I demonstrate that existing government guidance is open to misinterpretation and could result in erroneous data. Rather than forcing companies to do their own calculations, I would have thought it would be easy to promote free spreadsheets to allow organisations to calculate their gender pay gaps. I spent two hours producing this spreadsheet which you are free to download and test. As this is free, I offer no guarantees so please do contact me if you think any of the calculations are not correct.

Gender Pay Gap Calculator v1.0

There is a HELP sheet you can read in this file. To use it, enter your employee data in the yellow columns on the RAWDATA sheet. The calculations are done on the CALCULATIONS sheet and the results (which can be uploaded) appear on the DATATOREPORT sheet. I have also added a DATATOREPORTALT sheet which presents the results in a different format, using my suggestions as listed in this article. Should the government adopt my recommendations then this is the sheet that will contain the data to be uploaded.

RECOMMENDATION

The government should add links to free calculators of the gender pay gap reporting requirements. Better still they should allow the results to be uploaded using a text file, the format of which should be clearly stated on the government’s website so that developers know how to code their tools to automatically create this text file and upload it to the government’s portal.

8 – Organisations who employ less than 100 women should be flagged.

I have written in some depth about how the laws of chance can lead to significant gender pay gaps even when an organisation does not discriminate. The outcome of this work was this table which shows the upper and lower limits for the median woman’s hourly earnings that can occur by chance alone. These limits vary by the size of the employer and the % of employees that are women. Most notably, as the proportion of women in a company falls, the limits widen.

This means if a company employs only a few women, then it will be hard to draw conclusions about whether the pay gap is due to chance or some form of discrimination. Some smaller companies could find themselves unjustly accused of discriminating against women when they have merely been the victim of chance. It can be shown mathematically that the largest uncertainties occur when the number of women in an organisation falls below 100, regardless of how large the employer.

To avoid the issues I raise here, I think organisations who employ less than 100 women (or indeed 100 men, since my argument works the other way) should be flagged clearly in the government database. This would have two benefits. First any such flagged organisation should be given greater benefit of the doubt when looking at their gender pay gaps. Secondly, if the organisation is a large one, say 2000 employees, then being flagged for having less than 100 women would be a clear signal that the organisation is probably discriminating against women at the recruitment stage and could be challenged on that basis instead.

RECOMMENDATION

Any organisation who reports employing less than 100 women, should be clearly flagged on the government’s database. A caution should be displayed that such organisations will have unreliable gender pay gap figures.

9 – Publish both years’ results side by side on the government website.

Now that the 2017 round of gender pay gaps have been published, organisations can now upload their 2018 data. So far, 169 employers have done so, since the 2018 results are based on wages paid in April 2018. Looking at the 2018 data, I should comment that some organisations appear to have simply repeated their 2017 data in 2018 such as Adecco UK and others appear to have entered their 2017 results on the 2018 database by mistake such as Liverpool Airport Ltd.

Whatever the issues with the current 2018 data, I do think we should start to think about how to compare 2018 versus 2017 for an organisation. I propose to keep it simple to begin with and I recommend that the government’s website should simply display two sets of data and charts side by side. This allows one to see what has changed from one year to the next. For example, an organisation might have increased its median woman’s hourly earnings from £0.90 to £1.00 which would sound good. Then you look at the gender splits by income quartiles and you see that whereas in 2017, 75% of employees in the lower income quartile were women but in 2018, this has now fallen to 45%. This would raise the suspicion that the organisation has eliminated its median gender pay gap by sacking or outsourcing its lowest paid women.

RECOMMENDATION

The government should give some thought on how to present year on year comparisons for an organisation on the government’s website.

10 – Allow an organisation’s gender pay gap report to be rated

The government’s website allows organisations to upload its own analysis of its gender pay gaps. Looking at some of these, I can see that some organisations have clearly given thought on how to layout their data and looked into why they have large pay gaps. Two good examples are Ryanair and EasyJet. Others look like they have just filled in some standard bureaucratic template and give every indication they are not interested in their gender pay gaps.

Why has the government compelled all organisations employing more than 250 employees to report their gender pay gaps? I think the main reason is to force companies to examine their data to see if there are unexpected obstacles and issues impeding pay of women. If this is the case, then organisations that do this well should be highlighted and praised and those who are not interested could be named and shamed. At this point in time, I do not have a perfect answer to this issue but one thought I did have to allow organisations uploading their own reports to say that they are happy for their reports to appear on a separate website where anybody who reads them can give them a rating. This approach has two virtues, companies who elect not to do this can be challenged as to why not and those that do so can receive kudos for doing so and in doing so, the ratings may highlight some common themes among those doing well and those doing badly.

RECOMMENDATION

The government needs to remind the public why they have compelled organisations to publish their gender pay gaps and to find a way to publicise and highlight those organisations who are deemed to be successful. Obviously this will need a definition of what success looks like!

11 – Do not lower the minimum threshold of 250 employees.

I am aware of suggestions that the government may decide to lower the reporting threshold from 250 employees to 150 employees. I am against this proposal for the reasons I have laid out in suggestion #8 which recommends that organisations with less than 100 women (or men) are flagged. The reason why I recommend these organisations are flagged is that when this happens, the gender pay gaps become unreliable in the presence of the laws of chance. In other words, organisations that do not discriminate in any shape or form can expect to have large pay gaps purely by chance if they are a small organisation. If the government goes ahead with this step, this will simply exacerbate this issue and undermine trust in the whole gender pay gap reporting exercise.

RECOMMENDATION

The government should keep the minimum reporting threshold at 250 employees and implement suggestion #8 instead.

12 – Use a different process to measure racial & disability pay gaps

I am also aware of suggestions that the government should expand diversity reporting to cover race and disability pay gaps. Whilst, I concur with the motives for doing this, I have to state I am completely opposed to measuring such pay gaps by simply repeating the gender pay gap reporting process. If the government wishes to make such measurements, they should consider different approaches such as collating such data at an industry or local authority level.

There are two reasons why I oppose this. The first is that people who are not white or are disabled are minorities in the UK whereas women are not a minority. The second is that the proportion of the population that are women is very similar as one travels around the UK which is not the case for disabilities and ethnic minorities whose proportions of a local area can vary considerably from location to location. I repeat the chart I showed in suggestion #8 which shows that when the proportion of women in organisation falls to 10% or less, one can expect very large pay gaps purely by chance. If you substitute women for ethnic minorities or disabled people, you will still have the same problem in that you can expect large pay gaps purely by chance. If you now factor in that the expected proportion of staff that are disabled or non-white will depend on where the organisation is located, you will create additional chance variation which will widen the limits shown in the table.

RECOMMENDATION

If the government wants to measure racial and disability pay gaps, they should not replicate the gender pay gap reporting process. Alternatives approaches need to be considered such as reporting by industry bodies.