“Graduates aren’t skilled enough!” says a BBC headline. What is your immediate reaction? If you decide to find out more and read the article, you will see the following.
- A brief reference to a survey of a 174 organisations, half of whom are apparently moaning graduate skills.
- 3 brief interviews with recent graduates asking what they wish they had learned before starting their job.
After reading this, do you feel that a case has been made that universities are slipping up? How much weight should you place on this article and the information it contains? One of the major problems with news these days is that we are bombarded with articles about so many things that it can difficult to sort the good from the bad, especially when articles are referring to data in one way or another. My Evidence Hierarchy provides a short cut to assess the usefulness of news articles and with a bit of practice, I hope the result will be less stress for you about what is going on in the world.
Let me explain what the evidence hierarchy is and how it can be used. I should make it cle
ar that my hierarchy is mostly intended for the article itself, not the underlying research on which the article is about. A perfectly good piece of research could be ruined by a bad article and it is the article I want help you with.
The Evidence Hierarchy consists of 5 stages as shown in the graphic. I will start from the bottom.
- Beliefs – No evidence is described in the article. All that is written is someone’s belief that something is true.
- Anecdotes – Also known as case studies, this is a favourite of the media. Instead of reviewing the evidence, the writer of the article prefers to use some anecdotes as illustration. The graduate skills story I refer to at the beginning is a classic of this genre. The writer had a whole survey he or she could have used and virtually ignored it in favour of a quick vox pop. By definition, anecdotes are unstructured and generally biased and a poor way of coming to a conclusion.
- Observations – Instead of reporting a bunch a random anecdotes, the article looks at data collected in a more systematic fashion. The two most common forms of systematic observations are time series where data is collected at regular intervals and surveys where data is collected across a population at a specific point in time. The graduate skills article makes clear that a survey of 174 organisations was carried out but provides no details. Whilst much more structured and allowing more interpretation, observations can be biased if they fail to collect data from a representative population (e.g. an opinion poll that surveys too few Conservative voters) or if there are unknown/uncontrolled factors that end up distorting the data.
- Experiments – These seek to rectify the issues of observations by exerting more control over how the data is collected so that factors of interest are easier to see. For example, with graduate skills, you could recruit say 20 graduates from a university with a good reputation for skills training and 20 from another university with a reputation for ivory tower academic theory. Then you send 1 graduate from each university to a company which would result in 20 companies taking part. The companies would then report back on the graduates’ skills. If you find that employers complain more about the second university than the first, then it would be reasonable to conclude that the universities are the main driver of preparing graduates for work. If on the other hand, employers see both graduates as equally good or bad, this would suggest that it is the employers expectations and attitudes that are the main determinant of whether graduates are any good.
- Logic – This primarily occurs in Mathematics and it is rare to see an article which describes a logical reasoning and the implications of the logic but sometimes it happens outside of Maths. A good example was Einstein’s theory of relativity which was worked out through mathematical reasoning. In order to verify it though, an experiment was designed based on observations of the solar eclipse in 1919. These were undertaken by Sir Arthur Eddington on the island Principe and he showed that Einstein’s predictions were correct.
Let me illustrate the evidence hierarchy with an example. Of all things, I will choose the Loch Ness Monster and ask what evidence is there that it exists.
- Belief – I could simply say that the Loch Ness monster is real because I believe it to be real. Actually, I would love the Loch Ness monster to be real but this is not the same thing as believing it is real.
- Anecdotes – Many witnesses have claimed to have seen the monster along with various photos and videos showing something. Some have been shown to be fakes, some are implausible and some have yet to verified or debunked. Bear in mind though that a court of law relies on anecdotes i.e. witnesses but they are at least tested through cross examination but the Loch Ness monster witnesses have not been.
- Observations – a systematic sonar scan of Loch Ness in 1987 detected large moving objects underwater. Further scans have not always been able to verify this and various explanations have been put forward for these observations.
- Experiments – the best way to verify the existence of the Loch Ness monster is to catch one. One could set a trap with cameras and cages somewhere in the Loch. Jeremy Wade, presenter of River Monsters, put forward the fascinating hypothesis that the monster could be a Greenland Shark and went on to catch one off Norway. If he were to catch one in Loch Ness as well, that would be experimental verification of his hypothesis.
- Logic – I am not aware of any mathematical proof that the Loch Ness monster must exist!
You can see how the evidence hierarchy strengthens as you go from Belief to Logic. So when you next read an article about some recent research, try to identify which level of the hierarchy the research sits in and whether the article properly reflects. My future blog posts will give you plenty of examples to help you do this.
However, I want to point out two alternate viewpoints on my hierarchy.
The first alternate is that it is a not a hierarchy but a circle.
As someone whose first degree was in Mathematics, I know that whilst my subject might be the epitome of formal logic (which underlies all proofs), the dirty secret of Maths is that it relies on a set of beliefs known as axioms. An axiom could be 1+1=2 and from that we can use logic to derive various proofs. In practice, it is possible to prove 1+1=2 from even more fundamental axioms and in the early 20th century, mathematicians attempted to identify all the fundamental axioms of mathematics. Their goal was to identify a set of axioms (not unlike the 10 commandments) which would be complete and consistent. Completeness means that whatever problems are posed, the axioms allow an answer to be found. Consistency means that you would always get the same answer regardless of how you started and it would not be possible to use one axiom to prove 1+1=2 and another to prove 1+1=3.
Axioms underpin a vast range of human affairs including religion, politics, law, sport, etc where a set of rules or commandments have been agreed and then logic is used to make decisions based on the axioms. This is why we could talk about an evidence circle instead since logic follows on from beliefs and sometimes our logical decisions require experimental verification before we proceed. Unfortunately, history shows that human beings are not very good at creating complete and consistent axioms but there was always the understanding Mathematics was different. Kurt Godel had other ideas though and in 1931 (4 years after Heisenburg discovered his Uncertainty Principle in Quantum Mechanics) Godel published his two theories of incompleteness which proved that mathematicians could never be certain their axioms were complete and consistent. The axioms may well be but there is no way to prove this which was a startling realisation for mathematicians.
The second alternate viewpoint is that we should not be thinking that beliefs are always bad and experiments are always good. Consider this question “Does your partner love you?” If you answer yes, then I might ask “how do you know?” At that point, you might start recounting various anecdotes of times your partner demonstrated their love. If that wasn’t sufficient, you might start keeping a diary documenting your partner’s behaviour and how they treat you. If that didn’t satisfy you, you might decide to engage a really sexy charming person to chat up your partner and see if they can seduce them. When you find out that your partner didn’t rise to the bait, you finally realise that your partner loves you! At that point, you tell your partner “I’ve been keeping a diary of your behaviour and I tested you with that hot date and you passed! I have proof you love me!!” The next day, your partner announces that they are leaving you and quite right too.
What was the error? The answer is that when it comes to love, your belief should be sufficient and if you have to resort to proof then that is very likely to be a sign that love isn’t there. I like to use this example to finish my discussion of the evidence hierarchy. My key point is that we need to be honest with where we are on the hierarchy when we seek to put arguments forward. There is nothing wrong with using beliefs as your arguments, what is wrong is fooling yourself into thinking you have observational evidence say when in fact you have nothing of the kind. The MMR and autism scandal was a classic example of data that was nothing more than a mixture of Belief & Anecdote being passed off as Observation & Experiment. My reaction to that was if only journalists and readers had a better understanding of the evidence hierarchy/circle, some of the issues might have been avoided. By using my blog to post examples and which of the 5 boxes they sit in, my hope is that you will be better able to discriminate between articles in the future.
