How Not to Fall for Bad Statistics - with Jennifer Rogers

The Royal Institution
7 Aug 201942:20
EducationalLearning
32 Likes 10 Comments

TLDRIn this talk, the speaker discusses how to interpret risk in everyday life, debunking common misconceptions and providing a toolbox for understanding headlines about health risks. They explore the difference between relative and absolute risks, the importance of considering confounding factors, and the misuse of statistics in advertising. The speaker uses humor and real-life examples to make complex statistical concepts accessible and engaging.

Takeaways
  • πŸ“° The media often presents risk information in a way that can be misleading, focusing on relative risks rather than absolute risks, which can distort the true level of danger.
  • πŸ¦„ When evaluating risk, it's crucial to consider the actual numbers and the context in which they are presented, rather than just the sensational headlines.
  • 🐊 In understanding risk, public perception can be vastly different from the actual data. For example, more people die from hippos than crocodiles, contrary to what many might believe.
  • πŸš΄β€β™€οΈ The perceived risk of activities can be skewed by how they are measured. For instance, cycling is considered more dangerous than driving a car when measured by deaths per billion hours, but this doesn't account for other factors like distance traveled.
  • πŸ₯“ The impact of certain foods, like bacon, on health risks is often overstated in the media. It's important to look at the actual increase in risk (absolute risk) rather than just the percentage increase (relative risk).
  • πŸš— The effectiveness of interventions like speed cameras can be misinterpreted due to regression to the mean, where improvements are attributed to the intervention rather than natural fluctuations in data.
  • πŸ” When analyzing data, correlation does not necessarily imply causation. It's essential to consider other factors that might explain the observed relationship.
  • πŸ§€ Spurious correlations can be found in any dataset, and they can be humorous or misleading. It's important to question whether there is a true causal relationship or if other factors are at play.
  • πŸ† The concept of 'regression to the mean' is relevant in many areas, including sports, where temporary high or low performance can be misattributed to external factors like management changes.
  • πŸ™οΈ In comparing cities or regions for safety, such as London and New York, short-term statistics can be misleading. Long-term trends provide a more accurate picture of relative safety.
  • πŸ“Š Graphical representations of data can be manipulated to tell a specific story, sometimes at the expense of accuracy. It's important to scrutinize the scale and the actual numbers presented in graphs.
Q & A
  • What is the main topic of the talk?

    -The main topic of the talk is understanding risk and how to make sense of risk-related headlines.

  • Why are headlines about risk important for our daily lives?

    -Headlines about risk are important because they are supposed to inform our day-to-day decisions and help us live longer, healthier lives.

  • What is the first risk-related question the speaker asks the audience?

    -The first risk-related question the speaker asks is which animal is more dangerous: crocodiles or hippos.

  • According to the World Health Organisation, which animal causes more deaths, crocodiles or hippos?

    -According to the World Health Organisation, crocodiles cause more deaths than hippos, with 1,000 deaths a year compared to 500 from hippos.

  • What is the difference between relative risk and absolute risk?

    -Relative risk tells you the risk in one group compared to another, while absolute risk gives you the actual probability of an event occurring.

  • What is the example used in the talk to explain the difference between relative and absolute risk?

    -The example used is the risk of pancreatic cancer from eating bacon daily, which increases the relative risk by 20%, but in absolute terms, it means an increase from 5 to 6 cases in every 400 individuals.

  • What is the issue with comparing bacon and smoking as cancer risks based on statistical significance alone?

    -Comparing bacon and smoking based on statistical significance alone is misleading because it ignores the actual magnitude of the risks, which are vastly different.

  • What is the concept of 'regression to the mean' and how is it demonstrated in the talk?

    -Regression to the mean is the tendency for extreme results to be followed by more average results. It is demonstrated through a dice-rolling exercise where 'accidents' decrease after 'speed cameras' are placed, but this decrease is due to chance rather than the cameras.

  • Why did the speaker criticize the BBC's coverage of the story about living near a busy road and the risk of dementia?

    -The speaker criticized the BBC's coverage because it focused on a 7% increased risk of dementia from living near a busy road, while ignoring other factors like smoking and obesity, which have a much greater impact on dementia risk.

  • What is the issue with small sample sizes in surveys and how can it affect the reliability of the results?

    -Small sample sizes can lead to high uncertainty and unreliable results because they may not accurately represent the larger population. The confidence interval for the results may be wide, making it difficult to draw definitive conclusions.

  • How can graphics in advertisements or media sometimes mislead viewers about statistical data?

    -Graphics can mislead by using incorrect scales, presenting data in a misleading way, or not clearly showing the uncertainty in the data. This can result in viewers having a distorted understanding of the actual statistics.

  • What is the speaker's role in the Royal Statistical Society and what are they working on?

    -The speaker is a member of the Royal Statistical Society and is involved in a project aimed at improving data ethics in advertising.

Outlines
00:00
πŸ“° Understanding Risk in Daily Headlines

The speaker begins by discussing the prevalence of risk-related headlines in the media and their impact on public perception. They emphasize the importance of understanding how to interpret these risks, particularly the difference between relative and absolute risks. The speaker introduces the concept of a 'toolbox' for evaluating risk-related information and engages the audience with a survey on risky activities, highlighting common misconceptions about the dangers of animals, sports, and transportation modes.

05:01
πŸ– The Risky Business of Bacon

This paragraph delves into the controversy surrounding bacon and its alleged link to cancer. The speaker critiques the media's portrayal of bacon as a significant cancer risk, explaining the difference between relative and absolute risk. They use the example of pancreatic cancer to illustrate how a 20% increased risk in relative terms translates to a much smaller increase in absolute terms. The speaker also addresses the World Health Organization's classification of processed meats as carcinogenic, comparing it to the risk of smoking, and emphasizes the need for a more nuanced understanding of risk.

10:03
πŸš— Risk Perception and Measurement

The speaker continues the discussion on risk by examining how it is measured and perceived. They challenge the audience to think critically about the methods used to assess risk, using examples of cycling versus driving and the variability in risk assessment methods. The speaker also touches on the complexities of measuring risk in activities like flying, where the risk is not constant throughout the journey. This section underscores the importance of considering the methodology behind risk assessments.

15:08
🧠 Dementia Risk and Living Near Busy Roads

In this paragraph, the speaker critiques a study that linked living near a busy road to an increased risk of dementia. They highlight the importance of considering confounding factors and the need for a comprehensive analysis of risk factors. The speaker points out that the study did not control for family history, which could significantly influence both the risk of dementia and the likelihood of living near a busy road. They also discuss the media's selective reporting of risk factors, urging the audience to question what information is being omitted.

20:08
🎲 Regression to the Mean in Accident Statistics

The speaker introduces the concept of regression to the mean using a dice-rolling demonstration. They explain how placing speed cameras in areas with high accident rates might lead to a reduction in accidents, which could be mistakenly attributed to the effectiveness of the cameras. The speaker argues that this reduction could simply be a random fluctuation, illustrating the need for a more thorough analysis over time to determine the true impact of interventions like speed cameras.

25:09
✈️ Air Travel Safety and Statistical Fluctuations

This paragraph discusses the perception of air travel safety, particularly in light of a year with no passenger jet crashes. The speaker argues that such a statistic could be misleading and that fluctuations in crash rates are expected. They caution against overreacting to single-year data and emphasize the need for a broader perspective when evaluating safety trends. The speaker also addresses the comparison of murder rates in London and New York, highlighting the importance of considering longer-term data.

30:11
πŸ”’ The Misuse of Statistics in Advertising

The speaker critiques the use of statistics in advertising, particularly in the context of a toothpaste advertisement. They explain the concept of uncertainty in statistical results and the importance of understanding the difference between probability theory and statistical inference. The speaker emphasizes the need for a clear understanding of what is being measured and the potential for variability in results, urging consumers to be skeptical of statistical claims in ads.

35:12
πŸ† Sports Statistics and the Illusion of Causation

In this paragraph, the speaker discusses the misuse of statistics in sports, particularly in the context of managerial changes and their perceived impact on team performance. They explain how regression to the mean can explain seemingly sudden improvements in performance, which may be attributed to new managers but are actually part of the team's natural fluctuation in performance. The speaker also addresses the 'Sports Illustrated curse,' illustrating how exceptional performance can be followed by a return to average performance.

40:13
πŸ† Misleading Graphics in Statistical Presentation

The speaker concludes by highlighting the importance of accurately presenting statistical information, particularly in graphics. They critique several examples of misleading or incorrect graphics, emphasizing the need for clarity and accuracy in data presentation. The speaker encourages the audience to be vigilant in interpreting statistical graphics, considering the scale, the source of the data, and the potential for misrepresentation.

Mindmap
Encouragement to Question Media Statistics
Importance of Critical Thinking in Risk Assessment
Ben and Jerry's Graphic Deception
Scale Manipulation in Graphs
Pie Chart Summation Error
Hypothesis Testing and Confidence Intervals
Uncertainty in Survey Results
Misleading Ad Statistics
Ryanair's Response and Public Reaction
Statistical Analysis of Ryanair's Claims
Random Seat Allocation vs. Middle Seat Bias
London Murder Rate vs. New York
Safest Year for Air Travel
Living Near Busy Roads and Dementia Risk
Spurious Correlations in Media
Correlation vs. Causation
Bacon and Cancer Risk
Dangerous Modes of Transport: Biking vs. Driving
Dangerous Sports: Baseball vs. Cheerleading
Dangerous Animals: Crocodiles vs. Hippos
Importance of Context in Risk Interpretation
Relative vs. Absolute Risk
Need for Understanding Risk Numbers
Daily Exposure to Risk Headlines
Conclusion
Graphic Misrepresentation
Data Ethics in Advertising
Case Study: Ryanair Seating Algorithm
Real-life Risk Misinterpretations
Misleading Statistics
Risk Scenarios
Risk Assessment Tools
Introduction
Risk Perception and Misinterpretation
Alert
Keywords
πŸ’‘Risk
Risk is the possibility of harm or loss. In the context of the video, it is central to understanding how we perceive and react to potential dangers in our daily lives. The speaker discusses how headlines and statistics can influence our perception of risk, such as the risk of cancer from eating bacon or the risk of accidents in different sports.
πŸ’‘Relative Risk
Relative risk is a measure of how much more likely an event is to occur in one group compared to another. The video emphasizes the importance of understanding relative risks in health-related headlines, such as the increased risk of pancreatic cancer from eating bacon. It's crucial because it only shows the change in risk compared to a baseline, not the actual risk levels.
πŸ’‘Absolute Risk
Absolute risk is the actual probability of an event occurring. The speaker uses the example of pancreatic cancer to explain how to convert relative risk into absolute risk, highlighting the need to know the baseline risk before interpreting increased risks from certain behaviors like eating bacon.
πŸ’‘Correlation vs. Causation
Correlation is a measure that expresses the extent to which two variables are linearly related, while causation implies a direct cause-and-effect relationship. The video uses humorous examples, like the correlation between fizzy drink consumption and teenage violence, to illustrate the danger of assuming causation from mere correlation.
πŸ’‘Confounding Factors
Confounding factors are variables that can cause or influence the relationship between the variables being studied. The speaker mentions how factors like family history can confound the relationship between living near a busy road and the risk of dementia, emphasizing the need to control for these factors in statistical analysis.
πŸ’‘Regression to the Mean
Regression to the mean is the phenomenon where extreme values in a dataset tend to be closer to the average in subsequent measurements. The video uses the example of speed cameras and their placement based on accident hotspots to demonstrate how this statistical concept can mislead interpretations of cause and effect.
πŸ’‘Statistical Significance
Statistical significance is a measure that indicates whether the results of a study are likely to be true in the general population. The speaker critiques the World Health Organization's classification of processed meat as a carcinogen, arguing that statistical significance does not necessarily imply a high risk.
πŸ’‘Uncertainty
Uncertainty in statistics refers to the inherent variability in data that can affect the reliability of results. The video discusses how uncertainty impacts the interpretation of survey results and the need for larger sample sizes to reduce this uncertainty and provide more reliable conclusions.
πŸ’‘Confidence Interval
A confidence interval is a range of values that is likely to contain the true value of a parameter based on a statistical model. The speaker explains how confidence intervals can help determine whether a result is statistically significant by comparing it to the null hypothesis.
πŸ’‘Data Ethics
Data ethics involves the responsible and fair handling of data, particularly in advertising and public communication. The video touches on the Royal Statistical Society's efforts to improve data ethics, highlighting the importance of transparency and accuracy in presenting statistical information.
πŸ’‘Hypothesis Testing
Hypothesis testing is a statistical method used to determine if there is enough evidence to support a claim. The video explains how to use hypothesis testing to evaluate whether observed results, like a high percentage of agreement in a survey, are statistically significant and not just due to chance.
Highlights

The speaker discusses the importance of understanding risk in everyday life and how to interpret risk-related headlines.

A tool box of questions to consider when evaluating risk headlines is introduced.

The audience is engaged in a survey to assess their understanding of risk, involving dangerous animals, sports, and transport modes.

Contrary to popular belief, hippos are more deadly than crocodiles, and cheerleading causes more accidents than baseball.

The distinction between relative and absolute risks is explained using the example of bacon consumption and pancreatic cancer risk.

A comparison of the risk of lung cancer from smoking versus pancreatic cancer from bacon shows a significant difference.

The concept of statistical significance is explained and its limitations in quantifying risk are highlighted.

The importance of considering daily habits and lifestyle factors when evaluating health risks is discussed.

The difference between correlation and causation is emphasized with humorous examples, such as fizzy drinks and teenage violence.

The use of confounding factors in statistical analysis is introduced to explain spurious correlations.

The concept of regression to the mean is demonstrated with a dice-rolling experiment to illustrate random fluctuations.

The impact of regression to the mean on the interpretation of speed camera effectiveness is discussed.

The speaker shares personal experiences of challenging misleading statistics in advertising with Ryanair.

The misuse of statistics in a toothpaste advertisement is critiqued, highlighting the issue of small sample sizes.

The importance of considering effect size alongside sample size in statistical analysis is explained.

The speaker concludes with advice on interpreting statistics in the media, emphasizing the need for critical thinking.

Transcripts
00:00

[MUSIC PLAYING]

00:06

They just weren't very good.

00:07

Anyway, here we go.

00:08

So "Living is a Risky Business," I'm

00:10

going to be talking today about risk.

00:12

We are bombarded every single day

00:16

with headlines full of all of these things

00:19

that we should and shouldn't be doing in order

00:22

to live a longer life.

00:24

You know, dementia, sitting too long may increase a middle

00:26

person's-- middle-aged person's risk.

00:29

And sometimes these even have numbers on them.

00:32

So an everyday painkiller doubles the risk

00:34

of heart attacks and strokes.

00:36

A child's risk of brain cancer triples after just two CT

00:40

scans.

00:41

And we're supposed to use these headlines to inform

00:45

our day-to-day lives and to help us make decisions as to how

00:49

we should be living our lives.

00:51

But how do we actually make sense of these numbers?

00:54

So throughout this talk, I'm going

00:55

to give you a little bit of a tool box

00:58

as to all of the things that you should be thinking about,

01:00

all of the questions that you should be asking yourselves,

01:03

when you see these sorts of headlines in the newspapers.

01:08

But first of all, I thought I would try and get

01:10

an idea as to how good you guys are at understanding risk.

01:15

Do you know what risky activities are?

01:18

So I'm going to do a bit of a survey.

01:20

I can't help it.

01:21

I'm a statistician.

01:21

We can't help but do surveys.

01:23

So I've got some different risk scenarios.

01:28

And I want you to tell me which is the most risky.

01:30

So the first one is I'm going to ask you which

01:33

is the most dangerous animal?

01:34

So which animal causes the most deaths?

01:37

So I'll show you both options.

01:38

And then I'm going to ask you to vote for me.

01:41

So is it crocodiles or is it hippos?

01:46

So if you think it's a crocodile, give me a cheer.

01:49

[CHEERS]

01:50

It's about half a dozen, dozen people think crocodile.

01:53

If you think it's a hippo, give me a cheer.

01:55

[LOUD CHEERS]

01:55

OK, that's overwhelmingly in favour of the hippo.

02:00

I can tell you it's not looking good because--

02:03

[LAUGHTER]

02:05

--crocodiles, according to the World Health Organisation,

02:09

cause 1,000 deaths a year compared with 500 from hippos.

02:13

OK, so I got two more.

02:15

Let's see if you can redeem yourself.

02:17

Which is the most dangerous sport?

02:20

So which causes the most accidents?

02:21

Is it baseball or is it cheerleading?

02:24

So give me a cheer if you think it's baseball?

02:27

[CHEER]

02:28

Give me a cheer if you think it's cheerleading.

02:30

[CHEER]

02:31

Now, it's a little bit more 50/50,

02:33

probably slightly in favour of the cheerleader.

02:36

So you have slightly redeemed yourself.

02:39

Cheerleading does cause more accidents than baseball.

02:42

OK, last one, which is the most dangerous mode of transport?

02:49

Is it riding a bike or driving a car?

02:54

So give me-- give me a cheer if it's riding a bike?

02:57

[CHEER]

02:58

Give me a cheer for driving a car.

03:00

[CHEER]

03:00

That's pretty 50-50 actually.

03:02

OK, let me do that one more time.

03:04

Riding a bike?

03:05

[CHEER]

03:05

Driving a car?

03:07

[CHEER]

03:07

I'd say that's about 50/50.

03:09

I can tell you that actually riding a bike

03:11

is more dangerous than driving a car, 550 deaths per billion

03:16

hours versus 130 deaths.

03:19

This is a really interesting one though,

03:22

because it raises all sorts of questions as to how

03:24

do we even measure risk?

03:26

I've chosen to measure this based on the amount of time

03:29

a person would spend doing each activity.

03:32

Some people may choose to think about the number of accidents

03:34

per miles travelled.

03:37

But cycling still comes out as more dangerous.

03:40

Some people might think to do this, the number of journeys

03:43

that you take.

03:44

If we think about other modes of transport,

03:46

such as flying in airplanes, the risk

03:50

isn't constant the whole time that you're up in an airplane.

03:52

It's more risky as you take off and as you land.

03:55

So even just how do we measure risk

03:59

is a really interesting question in itself.

04:03

So now we've established that you're, OK, at risk,

04:06

I think that's a fair assessment.

04:08

As I said, I want to talk to you about risks

04:10

that you see every day and give you a toolbox as to everything

04:13

you should be asking.

04:15

And I want to start out by talking about the humble bacon

04:18

sandwich.

04:19

[LAUGHTER]

04:20

Now, according to the headlines bacon

04:22

is one of the worst things you can be eating.

04:25

It causes all sorts of different types of cancer.

04:29

This headline here, say in "Daily fry-up boosts

04:32

cancer risk by 20%."

04:34

So if you eat bacon on a daily basis,

04:36

you increase your risk of pancreatic cancer by 20%.

04:40

And that's a shocking statistic, that actually

04:43

caused bacon sales to plummet.

04:45

Is it really though something that

04:47

we need to be worried about?

04:49

Crucially, when we see headlines like this,

04:53

they're actually giving us what we call relative risks.

04:56

They're only telling us what the risk is in one group

04:59

relative to another.

05:00

So I know that if I eat bacon, I've

05:02

got a 20% increased risk compared

05:05

to those who don't eat bacon.

05:07

But I don't know anything about what the risk actually is.

05:12

And that's where absolute risks come in.

05:14

So absolute risks depend on the numbers

05:17

that are associated with each of these things.

05:21

So how do I take a relative risk and turn it

05:23

into an absolute risk?

05:24

First of all, I need to know what my chances are

05:27

of getting pancreatic cancer.

05:28

And according to Cancer Research UK, we have a 1

05:31

in 80 lifetime risk of pancreatic cancer.

05:35

So what does that mean?

05:36

That means if we were to take 400 individuals who

05:40

didn't eat bacon, we would expect five of them

05:44

to get pancreatic cancer anyway.

05:47

My flicker has decided to stop working.

05:51

So if we then look back at our headline,

05:54

our headline says that our daily fry-up boosts our cancer risk

05:58

by 20%.

06:01

20% is a fifth.

06:02

And what's a fifth of 5.

06:05

It's just one, meaning that our risk goes

06:07

from five in every 400 individuals

06:10

to six in every 400 individuals.

06:13

It's only an extra one person in every 400.

06:17

So whilst that 20% increase sounded really scary,

06:21

a headline that said it increases your risk

06:23

an extra one person in every 400 wouldn't sound anywhere near

06:27

as scary or anything.

06:30

You didn't need to be worried about it.

06:32

There was also a headline that said that bacon, ham,

06:35

and sausages were now as big a cancer threat as smoking,

06:38

the World Health Organisation were to warn.

06:41

Now, the reason for this is that the World Health Organisation

06:43

produces these lists of known risk factors

06:47

for different types of cancer.

06:49

And smoking was already on there as a known risk

06:52

factor for lung cancer.

06:54

And they were saying that because this processed meat was

06:56

now being added to that list for the first time,

06:59

it meant that they were as risky as each other.

07:03

Now, these lists are based on something called

07:06

statistical significance.

07:07

And statistical significance just

07:09

tells us whether or not something definitely

07:11

does or definitely doesn't cause cancer.

07:15

It doesn't quantify that risk in any way.

07:19

So how do the risks for smoking and lung cancer

07:22

compare to those risks that we've just

07:23

seen for bacon and pancreatic cancer?

07:26

So if we take our 400 individuals again,

07:29

if you got 400 people who don't smoke,

07:32

you would expect four of them to get lung cancer anyway.

07:37

If you smoke 25 or more cigarettes every single day,

07:40

that goes up 24 times, to 96 in every 400 individuals.

07:47

So that's an extra 92 in every 400

07:50

compared to that extra one in every 400

07:53

for the bacon and pancreatic cancer.

07:56

So, yes, they may both be statistically

07:59

significant in causing cancer.

08:02

But to say that they now were as big a cancer

08:04

threat as each other, they were as risky as each other,

08:07

is absolutely ludicrous because we

08:09

can see that there is a huge difference in the risks.

08:14

What we also need to think about when we see these headlines

08:17

is that it compared those who eat bacon every single day

08:21

with those who never eat it.

08:23

And the risk was only increased by an extra one

08:25

person in every 400.

08:27

If you only eat bacon, say, once a week

08:30

on a Saturday morning as a treat to yourself,

08:33

it's going to have an even smaller effect.

08:35

And so it's going to be absolutely tiny, this risk,

08:39

Plus, what you also need to think

08:40

about is if you're eating bacon for breakfast every day,

08:43

you're not eating fruit for breakfast every day.

08:46

You may be more likely to have an unhealthy lifestyle

08:49

in general.

08:50

And how do we know that it's the bacon that's actually

08:53

causing this increased risk of cancer,

08:55

and it isn't another one of these unhealthy lifestyle

08:58

factors instead?

09:00

And what we say in statistics is that correlation doesn't always

09:04

mean causation.

09:06

Here are some of my favourite headlines for demonstrating

09:10

correlation versus causation.

09:12

So, yeah, fizzy drinks make teenagers violent.

09:15

Fizzy drink a day make teenagers behave aggressively.

09:18

Children drinking fizzy drinks are regularly

09:20

more likely to carry a gun.

09:23

[LAUGHTER]

09:24

Now, it could be that drinking fizzy

09:28

drinks makes teenagers violent.

09:31

Or it could be that there's some other social demographic factor

09:35

that means a teenager is more likely to have

09:38

fizzy drinks in their diet.

09:40

And they're also more likely to be violent.

09:43

Or it could be that being violent is thirsty work.

09:46

And at the end of it, you want a fizzy drink.

09:48

[LAUGHTER]

09:49

We don't know which way around that relationship goes.

09:53

One of the best ways to think about correlation

09:56

versus causation is if you think about ice cream sales.

09:59

As ice cream sales go up, so do the number of drownings.

10:03

[LAUGHTER]

10:04

So does that mean that ice cream causes drownings?

10:10

Both of these things are affected

10:12

by something else, hot weather.

10:16

As the temperatures increase, we eat more ice cream,

10:20

as temperatures increase, we go into the sea more often,

10:22

meaning that there are naturally just more drownings.

10:26

Once we actually take that into account,

10:29

that direct relationship between ice creams and drownings

10:33

disappears.

10:34

We call this a confounding factor.

10:37

And once we account for the confounding factor

10:39

in our analysis, then that direct relationship

10:42

between these two things disappears.

10:44

There's a really nice website that I like to go on,

10:47

that allows you to correlate all these weird and wonderful

10:50

things with each other.

10:51

So I got on there.

10:52

And I've picked out some of my favourites.

10:54

If you google spurious correlations,

10:56

it's the first one that comes up.

10:57

You can have a lot of fun with it.

10:58

I spent too much time trying to find interesting correlations

11:01

to my presentations.

11:03

But there is a 98% correlation between the amount

11:07

of money spent on admission to spectator

11:09

sports and the number of people who died

11:10

by falling down the stairs.

11:13

So does this mean as we're all spending money

11:15

to go to these big arenas, we're all falling down

11:17

the stairs at the same time?

11:19

I don't know.

11:21

There is a 94% correlation between maths doctorates

11:24

awarded and the amount of money spent on pets.

11:27

Now, I'm a dog lover.

11:30

Am I a dog lover because I'm a mathematician?

11:33

I don't know.

11:34

My absolute favourite though is that there is a 95% correlation

11:40

between the per capita consumption of cheese

11:43

and the number of people who died by becoming tangled

11:46

in their bed sheets.

11:49

So does this mean that we shouldn't eat cheese before we

11:53

go to bed because we might die?

11:57

These are two things that are obviously

11:59

just-- they just happen to be correlated with each other.

12:02

And it doesn't mean that one is causing the other.

12:07

Now, you might be saying to yourself,

12:09

this is all well and good.

12:10

This is all very funny.

12:12

I know that cheese doesn't cause death by bedsheet.

12:15

When do I actually really need to think

12:17

about this in real life?

12:19

I was asked to comment on a story that was run by the BBC--

12:24

and it was January 2017, so just over two years ago now--

12:27

that said, live nearing a busy road increased

12:30

your risk of dementia.

12:31

It apparently increased your risk of dementia by 7%.

12:35

And the BBC got in touch and wanted me to comment on it.

12:37

And they wanted me to talk about this relative

12:39

versus absolute risk.

12:41

So I went and I had to look at the paper--

12:43

it was published in The Lancet--

12:45

just to try and get an idea as to what the absolute numbers

12:47

might have been.

12:49

And whilst I was looking at this study,

12:51

I realised that they hadn't controlled for family history

12:55

in their analysis.

12:57

And I argued that we know there's a huge family history

13:00

element to dementia.

13:03

But you could also argue that there's

13:04

a family history element as to where you might live.

13:07

If you grew up in the middle of the countryside,

13:10

you might be more likely to continue

13:11

living in the countryside as an adult.

13:13

If you grew up in a big city, you

13:15

might be more likely to live in a big city as an adult.

13:18

And so you've got this big family history

13:20

element to dementia and family history element

13:22

as to where you might live.

13:24

And I said the fact that that wasn't

13:25

accounted for in the analysis was a major let down

13:29

in the study.

13:31

Also while I was looking at this,

13:32

I looked at the supplementary material.

13:34

And it looked to all of these other things

13:37

that they had looked at, that might

13:39

be associated with dementia.

13:41

So this top row here are all of these different factors

13:44

that they thought might be associated with dementia.

13:48

So, for example, where you can see that smoking,

13:51

that 1.3 means that smoking increases your risk of dementia

13:56

by 30%.

13:57

The obesity, so obese versus a normal weight,

14:01

increases your risk of dementia by 64%.

14:05

And yet the newspapers had chosen

14:07

to really focus on this living near a busy road increasing

14:11

your risk of dementia by 7%.

14:15

And I said, you know, before you went

14:16

to pick up sticks and move to the countryside,

14:18

there are lots of other things that you

14:20

could do that would have a bigger

14:21

effect on your risk of dementia, quitting smoking,

14:25

losing weight, and higher education versus lower

14:28

education.

14:29

But the newspapers just chose not to report any of this.

14:32

And so what I would always say to you when

14:34

you see these headlines, have a little think to yourself,

14:37

what aren't they telling us?

14:38

What else could be going on?

14:41

So I'm asked quite a lot actually

14:43

to comment on stories that appear in the press.

14:47

And this was another one that I was

14:49

asked to comment on at the beginning of last year, that

14:52

said that 2017 was the safest year for air travel

14:55

as fatalities fall.

14:57

So in 2017, there were no deaths anywhere in the world

15:00

caused by passenger jet crashes.

15:03

And early into 2018, there then was a passenger jet crash.

15:07

And there was all this sort of investigation

15:10

as to has everything gone wrong?

15:12

Was 2017 the safest year we're ever going to have?

15:15

Do we now have to start investigating to try and figure

15:18

out what's happened?

15:20

And so I was asked to comment on this story.

15:23

And I want to do a little demonstration with you.

15:26

So there are dice in these two rows.

15:30

Some of you will have been given dice

15:31

or had dice underneath your seat as you sat down.

15:33

Could you just wave you dice subtly, please?

15:36

So we're going to do a little demonstration with these dice.

15:40

And we're going to do a little demonstration that

15:42

thinks about these things, speed cameras.

15:45

So everybody's favourites, I know.

15:48

So when speed cameras first came in,

15:51

the government needed to give some serious thought as

15:53

to where they might put them.

15:55

We couldn't put speed cameras absolutely everywhere.

15:58

Was there some sort of sensible strategy

16:00

that we could adopt, to decide where we might put those speed

16:03

cameras?

16:04

And we're going to recreate that exercise now.

16:07

And what the government decided to do

16:09

was to try and identify all the accident hotspots.

16:13

And they said that those were obviously

16:14

the most dangerous places.

16:16

And those were the ones that were in the highest need

16:19

of getting these speed cameras.

16:21

And we're going to recreate that now.

16:23

So all of you who have got the dice, in a second I'm

16:27

going to ask you to roll them.

16:28

I'm very aware of the fact that there's not that much room.

16:32

So can I just suggest you give them a good shake in your hand

16:35

and just drop them on the floor.

16:36

But, yes, so I'm going to ask you to do that.

16:38

I want you to do it twice.

16:40

And count up the score that you get.

16:41

And then we're going to decide where we're

16:43

going to put our speed cameras.

16:44

So if you could all roll them for me twice,

16:46

that would be marvellous.

16:51

There should be quite a few more on that row actually.

16:54

There should be more than one.

16:56

Oh, you've handed them back.

16:57

OK, that's fine.

16:58

OK, right, so did anybody get a 12?

17:04

Anybody get an 11?

17:07

Oh, I heard a twitch then.

17:09

Anybody get a 10?

17:11

I only got 1.

17:11

OK, right, what we do is we're going to redo this.

17:14

I usually have more dice.

17:16

I gave this talk last week to a group of teenagers.

17:19

And they stole my dice because they all

17:21

like to take souvenirs.

17:23

So I'm now doing a probability problem with fewer dice

17:26

than I normally would have.

17:28

So bear with me.

17:30

And we'll just repeat that again.

17:32

So if you could all just give them a really good roll

17:33

and repeat that.

17:34

Do it twice for me.

17:35

And we'll see what we get.

17:36

This is what happens when you present to teenagers.

17:39

They like souvenirs.

17:42

OK, right, how about this time.

17:44

Did anybody got any 12s?

17:47

Any 11s.

17:48

We've got two, OK, right, brilliant.

17:51

3?

17:51

3?

17:52

Sorry, it's the lights.

17:53

OK, so I'm going to give you a speed camera.

17:56

So we got a speed camera there.

17:57

You happen to be all spread out as much as you possibly

18:00

could be.

18:00

Oh, thank you very much.

18:02

And a speed camera over here.

18:03

Can I just ask you to pass that back behind?

18:05

That'll be brilliant.

18:06

Thank you.

18:07

So now we've got our speed cameras in place.

18:13

We now need to see if they've worked.

18:15

So what I want you to do is to all repeat the same thing

18:18

again.

18:19

Last time, I promise.

18:19

If you could all just roll your dice for me again, twice.

18:30

OK, where we have speed cameras, what did you get this Time

18:32

6.

18:33

6.

18:34

8.

18:34

8.

18:35

7.

18:36

7.

18:36

So we can see where we've got our speed

18:38

cameras we've seen a reduction in the number of accidents--

18:42

[LAUGHTER]

18:42

--meaning our speed cameras have worked right?

18:49

Maybe not.

18:51

OK, so this is a really nice demonstration

18:55

of what we call regression to the mean.

18:57

So what do I mean by regression to the mean?

18:59

Regression to the mean is we all behave according

19:02

to some sort of average.

19:03

But we don't get exactly the same value every single time.

19:07

We have these random fluctuations around it.

19:10

I simulated here a number of accidents,

19:13

where I kept the average a constant 10 all of the time.

19:18

But you see that I don't always get 10 accidents every month.

19:22

Sometimes I see higher than that.

19:23

Sometimes I see lower than that.

19:25

And this is what you would expect to see just by chance.

19:29

And crucially, it was one of these random highs

19:33

where I actually chose to put in my intervention.

19:38

You didn't have a higher chance of rolling

19:40

a high number that then changed when I gave you a yellow hat.

19:44

I chose to put the interventions in those places that

19:47

were randomly high the first time around.

19:51

And regression to the mean tells me

19:53

that I would have expected them to be lower the next time

19:56

around just by chance.

19:58

It's regressing to the mean.

20:00

So regression to the mean tells you

20:01

I've got something lower the first time around.

20:03

I expect it to be higher the next time around.

20:06

And this is exactly what happened

20:07

when speed cameras came in.

20:09

The government put them in those places

20:11

with the highest number of accidents.

20:12

They saw there was a reduction.

20:14

And then they said that the speed cameras

20:16

must have been working.

20:17

And it took a group of statisticians

20:18

to come in and say, actually, you

20:20

need to be looking at this over a long period of time

20:23

to be able to say whether or not the average is changing

20:27

through time, whether or not the average number of accidents

20:30

is actually decreasing.

20:31

So we see regression to the mean of the time in sports.

20:36

If you think about your favourite sports teams,

20:39

they'll go on random winning streaks.

20:40

And they'll go on random losing streaks.

20:43

When they go on losing streaks, they sometimes

20:45

sack their manager.

20:46

And a new manager comes in.

20:48

And they say, oh, look, they're winning again,

20:51

must be the manager.

20:52

A lot of the time, it can be explained by regression

20:55

to the mean.

20:56

That losing streak is just a random low.

20:59

And when they bring the new manager in,

21:01

they just go back to their average performance level.

21:04

And some research has shown that actually those teams that

21:06

stick with their managers see the bounce back

21:09

in form much quicker than those that

21:11

actually bring in new managers.

21:14

There's something called the Sports Illustrated curse, that

21:17

says when you appear on the cover of Sports Illustrated,

21:21

it's a curse.

21:21

You then go on to perform really badly.

21:25

But it can be explained by regression to the mean.

21:27

If you think about what does it take

21:29

to appear on the cover of Sports Illustrated,

21:31

you have to be at the very top of your game, which

21:34

is going to be a combination of your natural ability.

21:37

But you're probably also going to be riding one

21:39

of these random highs as well.

21:42

And this curse isn't necessarily a curse.

21:44

It's just you then--

21:46

that random high coming to an end.

21:48

And you're going back to your average ability.

21:51

So I argued that when we looked at this story here, all of this

21:55

could be explained by regression to the mean.

21:57

We would expect the number of air crashes and fatalities

22:01

to remain low.

22:03

But we are going to see these fluctuations around it.

22:05

Some years, we're just naturally going to see slightly more.

22:08

And some years we're just going to see slightly less.

22:11

And the fact that there have been none in 2017,

22:14

and then one in 2018, didn't necessarily

22:17

mean that everything had gone wrong.

22:19

And we all of a sudden needed to be having

22:21

these big investigations.

22:23

There were also stories about this time last year looking

22:27

at the London murder rate now beating New York,

22:29

as stabbings surged.

22:31

And there was a question as to whether or not

22:32

London was now a more dangerous city than New York.

22:36

And BBC's Reality Check actually looked into this.

22:39

So this is a really good resource,

22:41

that the BBC's Reality Check.

22:43

So they get statisticians to look at these sorts of claims.

22:46

And the claim was that London it overtake

22:48

overtaken New York for murders.

22:50

And it was now more dangerous.

22:52

And they found that a selective use

22:54

of statistics over a short period of time

22:57

appeared to bear it out.

22:58

But the reality was that New York still appeared

23:01

to be more violent than London.

23:02

If you looked at it over a longer period of time,

23:05

then New York did appear to be still more dangerous

23:08

than London.

23:11

So while we're on the topic of airplanes, I--

23:15

a vice president of the Royal Statistical Society.

23:18

And some of the work that I do with the RSS,

23:20

I have a bit of a hobby.

23:22

And I like to give Ryanair a headache.

23:25

[LAUGHTER]

23:29

So it first started out when I was

23:32

approached by BBC's Watchdog.

23:34

So Ryanair had changed their seating allocation algorithm.

23:39

It used to be that if you'd booked

23:41

as a group, when you checked in you would all

23:43

get to sit together.

23:45

And then they changed it and they

23:46

said if you didn't book seats together and pay for them,

23:49

you would be randomly scattered throughout the plane.

23:52

And loads of people started complaining to Watchdog,

23:55

saying that they thought there were too

23:57

many middle seats being given out.

23:59

So the window seat is quite desirable

24:01

because you get the nice view.

24:02

The aisle seat, you get a little bit of extra legroom.

24:05

The middle seat is seen as the least desirable seat.

24:07

But everyone seemed to be getting them.

24:09

And I thought that might be something going on there.

24:12

So they decided to send four of their researchers

24:16

on four flights.

24:17

And on every single one of the flights,

24:19

they were all allocated middle seats.

24:22

And they got in touch with me and said, hey,

24:25

what's the chances of that happening, if the seating

24:28

allocation is truly random?

24:30

So I did some stats for them.

24:32

And then I went on TV and I told them what I found.

24:35

So it wasn't actually a very complicated calculation

24:39

that we did.

24:39

They sent me the information available at the time

24:43

of check-in for each of the four flights.

24:45

So this is an example of one of them.

24:47

So when they checked into their flights

24:49

there were 23 window seats, 50 middle seats,

24:51

and 27 aisle seats available, so a total of 65 seats.

24:56

And using this, I can then work out the probability

24:59

that they're all given middle seats.

25:02

So the probability that the first person gets a middle seat

25:05

is 15 over 65 because there's 15 middle seats

25:09

and 65 seats in total.

25:12

The probability then that the next person

25:15

is given a middle seat is 14 over 64

25:18

because there's now 14 middle seats

25:19

available from 64 seats in total.

25:21

And I carry on.

25:22

So the probability of the third person is 13 over 63

25:25

and the fourth person is 12 over 62.

25:28

And if I multiply all of these together,

25:30

that gives me the probability of all four being middle seats.

25:34

And it's about 0.2%.

25:36

Which is one in 500, which isn't actually

25:40

that small a probability if you think about how many flights

25:42

Ryanair have every day.

25:44

One in 500 of this, it's not too surprising.

25:48

But this is just one flight.

25:50

As I said, they did it on another three flights.

25:53

And they all got middle seats on those three as well.

25:56

So I did the same calculations for the other three flights.

25:59

And then I combined it altogether.

26:01

And I found out the probability of all four researchers

26:03

get middle seats on all four flights was around 1

26:07

in 540 million.

26:10

So you were more than 10 times more likely

26:13

to win the national lottery than you were for this scenario

26:17

to happen.

26:18

But, you know, tiny probabilities

26:20

don't necessarily mean rare events.

26:22

So I went and had a look at Ryanair's facts and figures.

26:25

And they say that they only carry

26:27

130 million annual customers.

26:30

So I was pretty convinced that not

26:32

only was this a small probability,

26:35

it was a rare event.

26:36

And I was suspect as to whether or not

26:38

there was something going on with their algorithm.

26:41

Now, they said, you know, we've got our stats.

26:43

You've got yours.

26:44

My stats are right, thank you very much.

26:45

But anyway--

26:47

And it all kind of-- it kind of died a little bit of a death.

26:49

We got some media attention.

26:50

It was in the newspapers, but then not really very much

26:53

happened.

26:54

Until a couple of months later, when 12 women

26:57

all went on holiday together.

26:59

And they all got middle seats.

27:01

And they called the Telegraph.

27:03

And the Telegraph then called me and said, hey,

27:05

we heard you did some work on this.

27:07

What are the chances?

27:08

So I went through everything that I'd

27:10

done on the Watchdog story.

27:12

And they got in touch with Ryanair.

27:14

And at that point, Ryanair admitted

27:17

that they'd been lying.

27:19

They admitted that they actually kept window and aisle seats.

27:23

They held them back when randomly allocating the seats

27:26

because those were the ones that people

27:27

were most likely to pay for.

27:32

So this random allocation wasn't a random allocation

27:37

throughout the whole plane.

27:39

It was a random row within middle seats

27:42

that you were actually getting.

27:44

And so, yeah, I was really happy that I

27:46

managed to get Ryanair to admit to their customers

27:48

that they'd lied.

27:49

And I also managed to upset them in the process

27:52

because they didn't think that the negative media

27:54

attention, including the BBC investigation, was warranted.

27:58

So let's all feel sorry for Ryanair.

28:01

However, they are the gift that keeps on giving.

28:05

So last April, they released the results of a customer

28:11

satisfaction survey.

28:12

They said the 92% of their customers

28:14

were satisfied with their flight experience.

28:17

I thought, really?

28:19

I'd been on a Ryanair flight, 92%?

28:22

So I decided to take a look at the survey.

28:25

Now, bear in mind this was an opt-in survey.

28:29

So my argument was you're only going to opt into a survey

28:33

if you're really satisfied with your experience

28:36

and you want them to know about it

28:38

or you're dissatisfied with your experience

28:40

and you want them to know about it.

28:43

This was a survey that they asked people to fill in.

28:45

So this was the 92% here.

28:47

But if we look at the options that people

28:50

got when filling out this survey,

28:57

they went from excellent to OK.

29:00

So if you were dissatisfied with your Ryanair experience,

29:04

there was no way of expressing that dissatisfaction at all.

29:08

And I argued that you just then wouldn't carry out-- you just

29:11

wouldn't fill out the survey.

29:13

You'd just exit, switch it off, and you'd disappear.

29:16

So basically what you're asking was a group

29:18

of satisfied Ryanair customers just how satisfied they were

29:23

with their Ryanair experience?

29:24

And then we were really surprised

29:26

that once you combined three of the columns,

29:29

you got a high percentage.

29:31

So I went into the Times and I said as much.

29:34

And they had a really grown up response,

29:37

where they said 95% of Ryanair customers

29:39

haven't heard of the Royal Statistical Society.

29:41

[LAUGHTER]

29:42

97% don't care what they say.

29:45

And 100% said it sounds like their people need to book

29:47

a low-fare Ryanair holiday.

29:50

I mean, the stats in that are wrong

29:51

because if 100% say we need to book a low-fare holiday then

29:54

100% of them have heard of us.

29:56

So the stats are wrong to start off with.

29:59

But one of the members of the Royal Statistical Society

30:02

noted that there were 130 million annual Ryanair

30:06

customers.

30:07

And if 5% of them had heard of the Royal Statistical Society,

30:11

that meant that it was 6 and 1/2 million Ryanair customers who

30:14

had heard of the ROYAL Statistical Society.

30:16

And to be honest, we'd probably take that.

30:18

But there we go.

30:19

So, yeah, I like to--

30:21

I like to give Ryanair a headache as a hobby.

30:24

It's quite fun.

30:25

Interestingly, though, my boyfriend

30:27

is currently at the end of his training to be a pilot.

30:29

And Ryanair is one of the big options

30:31

that he might want to work for.

30:33

So that's irony right there.

30:36

So as I said, I'm a member of Royal Statistical Society.

30:39

And one of the big projects that we've got for this coming year

30:44

is actually trying to improve data ethics in advertising.

30:50

So why is this such an issue?

30:52

I'm going to play you a little advert.

30:54

And we're going to talk about adverts

30:55

in a little bit more detail.

30:57

[VIDEO PLAYBACK]

30:58

[MUSIC PLAYING]

31:01

- Pearl Drops cleans.

31:07

Pearl Drops whitens.

31:10

Pearl Drops protects.

31:13

Pearl Drops shines.

31:14

Pearl Drops 4D Whitening System, not only whitens, but cleans,

31:18

shrines, and protects, too.

31:20

Ultimate whitening, up to four shades

31:22

whiter in just three weeks.

31:23

Pearl Drops Tooth Polish, go beyond whitening.

31:27

[END PLAYBACK]

31:27

So we're used to seeing these all the time.

31:30

And we're to seeing these things all of the time,

31:34

at the bottom of them.

31:35

So there's some survey that's been done.

31:37

And so many people agree.

31:39

Now, there's lots of things wrong with this.

31:41

First of all, agree with what exactly?

31:43

I mean, there are a lot of claims in the advert.

31:45

It cleans, it whitens, it brightens.

31:48

Which of these exactly are they agreeing with?

31:51

But a lot of time when people hear I'm a statistician,

31:54

it's like, oh, adverts, can't trust anything, can you?

31:57

You can't trust any of the stats in there.

31:59

They use such small sample sizes that none of the results

32:03

are reliable.

32:04

And I want to talk about this in a little bit more detail.

32:07

So when you see this 52% of 52 people agreed,

32:12

what should you be thinking about when you see this?

32:14

So I want to talk a little bit about uncertainty.

32:16

What do I mean by uncertainty?

32:18

So if I was to take 10 people and line them up here and ask

32:21

them to flip a coin 10 times, I know

32:24

that there is a 50/50 chance of getting a head or a tail.

32:28

But if they all flipped it 10 times,

32:30

they wouldn't all get five heads and five tails.

32:33

Some people might get six heads.

32:34

Some people four heads, some people might get 10 heads.

32:38

That's what I mean by uncertainty.

32:40

In statistics, we talk about the difference

32:43

between probability theory and statistical inference.

32:47

So in probability theory, we know

32:49

the underlying probability.

32:51

And yet we see noisy data when we do experiments.

32:55

So that coin example, I know the underlying probability

32:58

is 50/50.

32:59

But I see noisy data when I do different experiments with it.

33:03

A lot of the time in statistics what we're actually trying

33:05

to do is to go the other way.

33:08

And we're trying to use samples of data

33:11

that we know are noisy and subject to uncertainty

33:15

and use that to tell me something about what

33:18

the underlying probability is.

33:21

So there I had 52% of 52 people agreed.

33:25

If I'd taken a different 52 people,

33:28

I wouldn't have seen exactly 52% agree.

33:31

And I'd have seen a slightly different number.

33:33

And a different 52 people would have given me

33:35

a slightly different number again.

33:37

And ultimately, what I'm trying to do

33:38

in statistics is to take that sample

33:41

and that piece of data that I know

33:43

is noisy and subject to uncertainty,

33:46

and use that to tell me something

33:47

about what the underlying probability is on a population

33:51

level.

33:52

So what I'm trying to do is I'm really

33:54

trying to create a hypothesis test to see whether or not

33:58

that percentage that I'm seeing is statistically significant?

34:02

So what do I mean in hypothesis testing,

34:04

how do I carry that out?

34:05

What I do is I formulate what we call a null hypothesis.

34:09

And a null hypothesis would be that the observations

34:12

are a result of pure chance.

34:14

So my underlying probability of people agreeing

34:17

is actually just 50/50.

34:20

It's all just down to chance.

34:22

And what I then say is let's assume that that's true.

34:25

Let's assume a null hypothesis is true.

34:27

Let's assume that the data I'm seeing is just random.

34:29

And it's just by chance.

34:31

What then is the probability of me seeing the data that I've

34:35

seen or seeing something at least as extreme

34:38

as what I've seen?

34:40

So let's break that down.

34:41

I understand that's quite a lot to get your head around.

34:42

So let's break that down for this particular example.

34:45

So my null hypothesis in this example would actually be 50%.

34:49

I'm assuming that these people have got a survey that say, you

34:52

agree that this toothpaste whiten your teeth, yes or no?

34:58

What pure random would be if they just randomly

35:01

ticked yes or no.

35:03

And so across my sample, I would expect

35:05

it to be about half yeses and half nos.

35:08

That would be what would be my pure random just by chance.

35:12

So 50% would be what would correspond to my pure chance.

35:17

So then if I had, going in the direction of agreeing or going

35:20

in the direction of disagreeing, that's

35:22

actually giving me information.

35:24

That's telling me some of the people

35:25

have got an opinion as to whether or not they disagree

35:28

or agree with that statement.

35:31

And I've got my 52% here.

35:34

And I know that that is subject to uncertainty.

35:36

So as I said, I know that if I took a different 52 people,

35:39

I'd get something slightly different from this.

35:42

So what I can do is I can put a confidence interval on this.

35:45

And this confidence interval is related to the sample size.

35:49

And it tells me, OK, 52%, that's the best

35:52

estimate as to what the true underlying probability might

35:56

be.

35:56

But what could it be?

35:58

What values could it possibly take?

36:00

And a confidence interval then gives me

36:01

a range of values that might be plausible.

36:05

And as I increase my sample size,

36:08

I actually decrease the amount of uncertainty.

36:11

And I decrease-- I make my confidence interval smaller.

36:16

And what we're looking for in hypothesis testing

36:18

for a statistically significant result

36:20

is we want that confidence interval to not cross

36:24

the null hypothesis.

36:25

So my null hypothesis here was 50%.

36:28

I want my confidence interval to not cross that 50%.

36:33

If it doesn't cross that 50%.

36:34

I say it's a statistically significant result.

36:37

If it does cross that, then I say

36:39

I haven't got enough evidence to say, actually

36:40

that it's just pure chance.

36:42

It could just be pure chance.

36:45

So that confidence interval really

36:47

matters when I've got something close to the null hypothesis

36:51

because that 52% is really close to that 50%.

36:55

I'm going to need quite a big sample

36:57

size to be able to make that confidence interval small

37:00

enough so that it doesn't cross that 50%.

37:04

If, on the other hand though, I had

37:06

to get a result that was a lot further away from that 50%,

37:10

I wouldn't necessarily need to have as big a sample size

37:14

because it's not as close to that null hypothesis of 50%.

37:18

It doesn't matter if the confidence interval is wider.

37:22

So yes, when you see these surveys

37:25

that are being done on small samples,

37:28

it's not always a problem.

37:30

It depends on how big your result actually is.

37:33

It's not just the sample size in itself,

37:36

but it's also what we say, you know, the effect sizes as well.

37:39

So just back to our example, we have 52% of 52 people agreed.

37:44

A 95% confidence interval on this based on these 52 people

37:48

is 38 to 66.

37:51

So 95% means if I was to repeat this a hundred times,

37:54

I would expect 95% of them to between 38 and 66.

37:58

And it crosses that 50% mark.

38:00

So here, this is no different from just pure randomness.

38:03

This is no different from people just flipping a coin,

38:06

saying yes or no, I agree with that statement.

38:09

If I was to take another one that said 74% of 54 men

38:13

agreed with some statement after 28 days,

38:16

the confidence interval on this is 60 to 85.

38:20

So it's a similar kind of sample size.

38:22

We've got similar sized confidence interval.

38:24

But because our treatment effect was 74% to start off with,

38:27

and that's a lot further away than the 50,

38:30

I have enough evidence here to say that there is a difference.

38:34

And actually people do have a preference.

38:36

And I'm just going to finish off now

38:38

with a couple of final graphics, that just say-- because I don't

38:42

think uncertainty necessarily has to be

38:44

very difficult to communicate.

38:46

I think--

38:47

[LAUGHTER]

38:49

When we look at the weather and we

38:51

look at when they tell you your probability of rain,

38:54

I mean, these numbers are ridiculous.

38:56

So what is it?

38:57

I mean, at 3 o'clock we have a 10% chance of rain.

39:00

That goes up to 13% at 4 o'clock and 16% at 5 o'clock.

39:05

What am I supposed to do with this information?

39:08

I don't know what the uncertainty is on that.

39:10

And they're really precise point estimates.

39:13

But it would be super-easy to communicate the uncertainty

39:17

using some sort of graphic.

39:19

Now, graphics have the ability to do great good.

39:23

They also do have the ability to do great evil.

39:26

And I just want to finish off with a couple of my favourite

39:29

bad graphics because it is something you really need

39:32

to watch out for when you're looking at stats in the media.

39:35

So there's this one, which is one of my favourites.

39:38

This is the presidential run.

39:39

So a pie chart should sum up to 100%.

39:43

[LAUGHTER]

39:45

This doesn't.

39:46

And they've obviously here asked would you back this person,

39:50

yes or no?

39:51

And then thought that a pie chart

39:53

was the most appropriate way to communicate that information.

39:57

This one, I've got no idea what they asked.

40:00

Half of Americans have tried marijuana today?

40:03

I'm not-- I don't know if I believe that.

40:05

But if 43% of them have tried it in the last year, which

40:09

includes today, how have 51% tried it?

40:13

The numbers are all wrong.

40:14

I can't figure out what's going on.

40:16

They have however, though, included uncertainty.

40:19

We know it's plus or minus 4%.

40:23

But I've got no--

40:24

I've got no idea.

40:25

This one from the Office of National Statistics

40:27

is a very sneaky one.

40:28

And one that shows you always need

40:30

to look at the scale of the plot because actually this

40:33

is a increase in GDP.

40:36

And it went from 0.6% to 0.7%, they're

40:44

predicted growth upgrade.

40:46

And it's a 0.1% increase.

40:51

And that looks a lot bigger on that plot.

40:54

And if we look at the axis along the bottom--

40:57

and look at the scale of that.

41:00

If you zoomed out onto that plot and looked

41:02

at the whole percentage line, it would

41:04

be just a minuscule difference.

41:06

So look at the scale.

41:07

And my last one is my favourite, from Ben and Jerry's.

41:10

I don't know what world we live in where 62 is smaller than 61.

41:14

But I don't want to live in that world.

41:17

Here is an example where Ben and Jerry's have got

41:19

a story that they want to tell.

41:22

And the numbers didn't quite agree with that story.

41:25

So they decided to produce a graphic that

41:27

told that story anyway.

41:29

And hoped we wouldn't look at the numbers in enough detail.

41:32

Look at the size of that 24% compared to that 21%.

41:36

They've got a clear story that they wanted to tell.

41:38

And if you were just flicking through a magazine,

41:40

you might not necessarily look at the numbers

41:42

in as much detail.

41:44

So yes, so very, very naughty from Ben and Jerry's.

41:46

So if you look at stats in the media,

41:49

I would encourage you think, relative

41:51

versus absolute risks, correlation versus causation.

41:55

Could this have happened just by chance, this regression

41:58

to the mean?

42:00

Yeah, eat bacon.

42:02

But don't eat cheese before you go to bed.

42:04

Thank you very much.

42:06

[APPLAUSE]