{"id":2450,"date":"2020-04-02T12:33:05","date_gmt":"2020-04-02T11:33:05","guid":{"rendered":"https:\/\/marriott-stats.com\/nigels-blog\/?p=2450"},"modified":"2020-07-18T10:38:09","modified_gmt":"2020-07-18T09:38:09","slug":"be-more-accurate-with-a-smaller-sample-size","status":"publish","type":"post","link":"https:\/\/marriott-stats.com\/nigels-blog\/be-more-accurate-with-a-smaller-sample-size\/","title":{"rendered":"Coronavirus #2 – What sample size does Boris need to lift all restrictions?"},"content":{"rendered":"

April Fools day 2020 saw the hive mind of social media asking what the sample size should be to measure the extent of the Coronavirus in the UK.\u00a0 I could see that many people responding were reaching for standard methodologies which are usually are based on specifying a desired confidence interval.\u00a0 In doing so, they were overlooking a much more effective and relevant alternative based on the methodology of Acceptance Sampling<\/a>, <\/strong>first developed by the US Military in World War 2<\/span>.<\/strong><\/span><\/p>\n

<\/p>\n

Data, Evidence, Decisions<\/span><\/strong><\/h4>\n

This is the strapline of the Royal Statistical Society<\/a> of which I am a member.\u00a0 It represents the order in which a lot of statistical analysis is done.\u00a0 First, collect the Data<\/strong><\/span>, then analyse it and extract the Evidence<\/strong><\/span> (or Insights<\/strong><\/span> which is my preferred word) and then, based on the evidence,\u00a0make Decisions<\/strong><\/span>.<\/p>\n

It also works in reverse.\u00a0 You have a Decision<\/strong><\/span> to make.\u00a0 From that you work back to identify the Evidence<\/strong><\/span> you need to make the decision.\u00a0 From the required evidence, you work back to identify the Data<\/strong><\/span> that is capable of giving you the evidence.<\/p>\n

If Boris Johnson wants to commission a survey to understand what people think of his handling of the Coronavirus pandemic, the flow Data > Evidence > Decisions<\/strong><\/span> is the right one to use to estimate the sample size of the survey.\u00a0 You start by asking “How accurately do we wish to measure the answers to the questions?<\/em>” and the answer activates the Data<\/strong><\/span> step.\u00a0 Once the data is collected, it can be turned into Insights<\/strong><\/span> and then Decisions<\/strong><\/span>.<\/p>\n

As of today, the UK population is largely locked down, nearly 2,500 have died of (or with) Covid19 and the economy is in recession.\u00a0 The questions Boris wants answered above all else are “Can I lift all or part of the restrictions today?<\/em>” and “If not today, then when?<\/em>“.\u00a0 Both require the reverse flow Decisions > Evidence > Data<\/strong><\/span>.\u00a0 Given the various Decisions<\/strong><\/span> that Boris can take, his advisors can identify the Evidence<\/strong><\/span> needed to allow those decisions to be made and that evidence will determine what Data<\/strong><\/span> is needed.<\/p>\n

Boris in the cockpit<\/strong><\/span><\/h4>\n

The first part of any flight is Take Off.\u00a0 After receiving clearance to take off, the pilot open the throttles and the plane starts to accelerate down the runway.\u00a0 During Take Off, the pilot has the 3 key decision points to pass through V1 – Rotate – V2<\/span>.\u00a0 <\/strong>Each denote an aircraft speed threshold above which the pilot is committed to certain courses of actions and other courses of actions are closed off.\u00a0 \u00a0Specifically,<\/span><\/p>\n

    \n
  1. V1<\/strong><\/span> – above this, the aircraft is committed to take off since the aircraft will not have enough runway to stop should take off be aborted.<\/li>\n
  2. Rotate<\/strong><\/span> – the speed at which the pilot pulls back on the stick to launch the aircraft into the air.<\/li>\n
  3. V2<\/strong><\/span> – above this, the aircraft is going fast enough to climb away, raise its undercarriage and flaps and fly away to its destination.<\/li>\n<\/ol>\n

    Prior to V1, the pilot can abort take off and start all over again.\u00a0 After V2, the aircraft is flying safely and can settle down to its normal routine.\u00a0 In between, if a problem develops, it may not always be possible to resolve it safely.\u00a0 The tragic crash of the Air France Concorde in 2000<\/a> was an example.\u00a0 The aircraft had already passed V1 when its fuel tank was ruptured by debris on the runway and it caught fire.\u00a0 The aircraft had to take off but was unable to reach V2 and fly normally and crashed before it could make an emergency landing.<\/p>\n

    Boris is our pilot today who needs to get the aircraft UK Life and Economy flying again at some point.\u00a0 Before he can open the throttles and start take off, he needs to know his Decision <\/strong><\/span>points V1 & V2.\u00a0 His advisors can analyse the medical and economic data and tell him what Evidence<\/strong><\/span> constitutes V1 & V2.\u00a0 Now all he needs is the Data<\/strong><\/span> to tell him (and us) whether the UK has reached V1 and lift some restrictions or whether we have reached V2 and can lift all restrictions.\u00a0 What Sample Size<\/strong><\/span> will give us the necessary data?<\/p>\n

    Calculating the Sample Size<\/span><\/strong><\/h4>\n

    Twitter wisdom at the moment is telling us to do mass testing of the population to find out if we are at V1 or V2 or even if we can start our take off run.\u00a0 Is this how pilots do it?\u00a0 A pilot of Airbus A380 will have over 500 passengers on board.\u00a0 Surely it would be good idea to get every passenger to use a stopwatch during the take off run and estimate when V1 has been reached and tell the pilot accordingly?\u00a0 After all 500 is a good sample size!<\/p>\n

    I sincerely hope no-one has gone through this experience.\u00a0 Instead the measurement is taken by a sample size of one (a single instrument though I assume there are backups) and is noted by a sample size of two, the two pilots, who will translate that into a decision.\u00a0 This points out a fundamental tension in trying to measure something accurately.\u00a0 The overall error is a combination of Sampling Error<\/strong> <\/span>(a function of sample size and design) and Measurement Error<\/strong> <\/span>(a function accuracy, precision & speed).<\/p>\n

    When it comes to decision making, especially those that have to be made under extreme stress of time or consequences, targeted & rapid sampling that answers a precise and demonstrably relevant question is far superior to large & slow samples that simply provides data to set of a vague questions.\u00a0 That is what Acceptance Sampling<\/strong><\/span> does.<\/p>\n

    A COVID19 example of Acceptance Sampling<\/strong><\/span><\/h4>\n

    Let’s use the following example.\u00a0 Antibody tests are in the news and have been described as a “game changer<\/em>” by the UK Chief Medical Officer.\u00a0 The reason is that if we can determine if an individual is immune to the disease using an antibody test, they can be released from restrictions and help get the UK flying again.\u00a0 If a certain percentage of the population is immune, then the population is said to have herd immunity.<\/p>\n

    Let’s use M<\/span><\/strong> to denote the % of the population that is immune to COVID19.\u00a0 How high does M<\/strong> have to be for ALL restrictions to be lifted?\u00a0 This is a question that Boris could plausibly ask of his experts.\u00a0 M<\/span><\/strong> will have to be at a level higher than the suspected herd immunity level (currently thought to be between 50-60%) both as a margin of safety but also to give an extra buffer for when international travel restarts and we start mixing again with populations with lower levels of immunity.\u00a0 I am going to set our target value for M to be 80% just to keep calculations simple.\u00a0 This makes 80% our V2 value, above which the UK is free and clear to fly and navigate.\u00a0 For V1, I am going to set this to be 50% (the minimum level for herd immunity) above which we can start to relax some restrictions but not all.<\/p>\n

    In the Acceptance Sampling<\/a> world, which are formalised by British Standards BS6000 (Numerical data) & BS6001 (Sampling by Attributes)<\/a> in the UK, V2 is called the Acceptable Quality Level (AQL)<\/strong><\/span> or Acceptable Outcome Level (AOL).\u00a0 V1 does not have a standardised name but I like to use the terms Unacceptable Quality Level (UQL)<\/strong><\/span> or Unacceptable Outcome Level (UOL).\u00a0 Also, acceptance sampling uses the language of defects or failures for each individual sample since it was initially developed by the US military initially as a way of inspecting ammunition batches to see if they were safe to dispatch to the troops.\u00a0 So the AQL for our COVID19 example is in fact 20% i.e. if less than 20% of the population is “defective” as in not immune, then that is an acceptable outcome.\u00a0 Similarly the UQL for COVID19 immunity is 50% i.e. if more than 50% of the population is not immune (“defective”) then that is an unacceptable outcome.\u00a0 Because of this switch from immune people to non-immune people, I will now refer to P<\/strong> instead of M<\/strong> where P<\/strong> = 100% – M<\/strong>.<\/p>\n

    An acceptance sampling plan consists of two elements denoted by R & N.<\/strong><\/span><\/p>\n

      \n
    1. N<\/span><\/strong> – the number of items\/people to be sampled and measured<\/li>\n
    2. R<\/strong><\/span> – the minimum number of items\/people who have to be deemed “defective” for the batch (or population) to be REJECTED<\/li>\n<\/ol>\n

      R & N create a decision rule which can be written as<\/p>\n