Marco Bonzanini - Lies, damned lies, and statistics

EuroPython Conference

6 Sept 201827:41

EducationalLearning

32 Likes 10 Comments

TLDRThe speaker discusses the use, misuse, and abuse of statistics in everyday life, emphasizing the importance of critical thinking when interpreting statistical data. They debunk common misconceptions, such as the confusion between correlation and causation, and introduce concepts like lurking variables and the Simpson's paradox. The talk also addresses various biases, including sampling bias and survivorship bias, and the potential for misleading data visualization. The speaker cautions about the misuse of statistical significance and p-values, and the dangers of data dredging. The presentation concludes with a call to always question the context and motivations behind statistical findings, advocating for a more informed and discerning approach to data analysis.

Takeaways

📊 **Understanding Statistics**: The speaker emphasizes the importance of comprehending statistics to be a well-informed citizen, not just for those with advanced degrees in math.
😄 **Humor in Data**: The presentation starts with a humorous example about the population density of Vatican City and the concept of one Pope at a time.
🔍 **Correlation vs. Causation**: A key point made is the need to differentiate between correlation, which indicates a relationship, and causation, which implies a direct cause and effect.
🔑 **Lurking Variables**: The concept of lurking variables is introduced to explain how a third, unseen variable can influence the correlation between two observed variables.
📈 **Simpson's Paradox**: The paradox is explained as a situation where data can be analyzed differently, leading to seemingly contradictory conclusions when grouped in various ways.
🤔 **Sampling Bias**: The audience is reminded of the potential for bias in sampling, which can lead to incorrect conclusions when a sample doesn't represent the larger population.
📉 **Survivorship Bias**: The issue of survivorship bias is highlighted, where focusing only on 'winners' can lead to an incomplete understanding of a situation.
📊 **Data Visualization**: The power of data visualization is discussed, showing how it can be used to communicate complex ideas simply and effectively.
📉 **Manipulation in Visualization**: The potential for manipulation in data visualization is also covered, warning against graphs that may distort the truth by starting axes at unconventional points.
⚖️ **Statistical Significance**: The distinction between statistical significance and practical importance is made clear, with the former not necessarily implying the latter.
🎣 **Data Dredging**: The practice of data dredging, or looking for patterns in data until something significant is found, is cautioned against due to the risk of false conclusions.
🤏 **P-Values and Data Fishing**: The concept of p-values is introduced, and the dangers of data fishing, where one might manipulate data to achieve a desired p-value, are discussed.

Q & A

What is the main focus of the talk?
-The main focus of the talk is the use, misuse, and abuse of statistics in everyday life. It aims to educate people on how not to be misled by statistical data and to be more discerning when interpreting statistical information.
What is the correlation mentioned in the talk?
-Correlation, as discussed in the talk, refers to a relationship or connection between two variables or events. It can be measured in terms of strength and is often visualized through a linear correlation, where one variable increases or decreases as the other does.
What is the difference between correlation and causation?
-Correlation indicates a relationship between two variables, but it does not imply that one variable causes the other to change. Causation implies a direct cause-and-effect relationship, which is a stronger claim than correlation.
What is a lurking variable?
-A lurking variable is a factor that is not immediately visible but influences the relationship between two other variables. It can help explain why seemingly correlated variables appear to have a connection that might not be directly causal.
Why is the Simpsons paradox mentioned in the context of data analysis?
-The Simpsons paradox is mentioned to illustrate how aggregated data can sometimes be misleading. It occurs when data is grouped differently, and the overall trend appears to reverse when looking at individual groups, highlighting the importance of considering data distribution.
What is sampling bias?
-Sampling bias is a systematic error that occurs when a subset of individuals is selected for a study in a way that is not representative of the larger population. This can lead to incorrect conclusions being drawn from the study.
What is survivorship bias?
-Survivorship bias is a type of sampling bias where only the 'winners' or survivors are considered, and the 'losers' or those who did not survive are ignored. This can lead to an overestimation of success rates and outcomes.
How can data visualization be misleading?
-Data visualization can be misleading if it is manipulated, such as by starting the axis at a value other than zero to exaggerate differences, or by omitting important context, which can lead to a misinterpretation of the data.
What is statistical significance?
-Statistical significance refers to the likelihood that the results of a study are not due to chance. A common threshold for statistical significance is a p-value of less than 0.05, indicating that there is a less than 5% chance that the results occurred by random chance.
What is data dredging?
-Data dredging, also known as data fishing, involves searching through data for patterns or relationships without a prior hypothesis. This can lead to false discoveries because the same data is used to both find and confirm the patterns.
Why is it important to question the context and funding of a study?
-Questioning the context and funding of a study is important because it helps to identify potential biases and vested interests that could influence the results. Understanding the bigger picture can help discern whether the presented data is being used accurately and ethically.

Outlines

00:00

😀 Understanding Statistics and Their Misuse

The speaker begins by humorously addressing the three types of lies: lies, damned lies, and statistics. They clarify a statistical fact about the population density of Popes in Vatican City, using it as a segue into a broader discussion about the everyday use and potential misinterpretation of statistics. The talk aims to educate on how to be critical consumers of statistical information without requiring advanced mathematical knowledge. The speaker also gauges the audience's familiarity with Python, highlighting its positive correlation with well-being.

05:04

🔍 Correlation vs. Causation and Lurking Variables

The speaker delves into the concept of correlation, emphasizing that it does not imply causation. They explain the difference between positive and negative linear correlation through examples, such as ice cream sales and drowning incidents. The importance of considering lurking variables that might explain the observed correlation is discussed, using the example of temperature affecting both ice cream sales and swimming activities.

10:05

📊 Data Visualization and Simpson's Paradox

The speaker discusses the power of data visualization in conveying complex concepts and its role in data analytics. They introduce the Simpson's paradox with an example from the University of California's admission rates, highlighting how aggregated data can sometimes be misleading. The paradox is explained by showing how men and women tend to apply to different departments with varying admission rates, which can skew the interpretation of the data when not analyzed at a granular level.

15:06

🤔 Sampling Bias and Its Impact on Data Interpretation

The speaker addresses the issue of sampling bias, using the historical example of the 'Dewey Defeats Truman' headline based on a biased survey. They also mention survivorship bias, a common pitfall when focusing only on successful cases while ignoring failures. The speaker underscores the importance of representative sampling to avoid skewed results and discusses how data visualization can be used to either clarify or obfuscate the truth, depending on the intent.

20:07

📉 Manipulation of Data Visualization and Statistical Significance

The speaker critiques the manipulation of data visualization for misleading purposes, using examples from politics and media. They discuss how the choice of axis scales can alter perceptions of data. The concept of statistical significance and its misinterpretation in everyday language is also explored, with a clarification that statistical significance does not equate to importance or usefulness. The misuse of p-values and the issue of data dredging are highlighted as common statistical pitfalls.

25:10

🧐 Questioning Data and Avoiding Bias

The speaker concludes by emphasizing the importance of questioning the data presented to us, especially when it aligns with a particular agenda. They caution against the blind acceptance of media headlines and the need for a deeper understanding of the context and methodology behind statistical studies. The speaker encourages critical thinking and a healthy skepticism towards data, advocating for a more informed approach to interpreting statistical information.

Mindmap

Keywords

💡Statistics

Statistics refers to the collection, analysis, interpretation, presentation, and organization of data. In the video, it is discussed as a tool that can be used, misused, or abused in everyday life. The video emphasizes the importance of understanding statistics to be a good citizen and to interpret data accurately, without being misled by potential biases or misrepresentations.

💡Correlation

Correlation is a statistical term that describes a relationship between two variables. The video uses the example of ice cream sales and drowning incidents to illustrate that correlation does not imply causation. It is a key concept in understanding how statistics can be misinterpreted to suggest a connection where none may exist.

💡Lurking Variable

A lurking variable is an unobserved or ignored factor that actually influences the relationship between two other variables. In the context of the video, the lurking variable in the ice cream sales example is the temperature, which affects both ice cream sales and the number of drownings, thus illustrating the need to identify and consider all relevant factors when analyzing data.

💡Simpson's Paradox

Simpson's Paradox is a phenomenon in probability and statistics where a trend appears in different groups of data but disappears or reverses when these groups are combined. The video uses this paradox to highlight how data can be sliced and diced to tell different stories, and the importance of looking at the bigger picture when analyzing statistics.

💡Sampling Bias

Sampling bias occurs when a sample of individuals is not representative of the larger population being studied. The video references the historical example of the 'Dewey Defeats Truman' headline, which was based on a biased sample of the population, illustrating the dangers of drawing conclusions from non-representative data.

💡Survivorship Bias

Survivorship bias is the error of focusing on the survivors or winners in a situation and overlooking those that did not survive or succeed. The video mentions this bias in the context of focusing on successful individuals like college dropout billionaires, which can lead to an overestimation of the likelihood of success for others who follow the same path.

💡Data Visualization

Data visualization is the presentation of data in a graphical format. The video discusses how visualization can provide better insights into a dataset and also how it can be used to mislead or emphasize certain findings. Examples include charts that start from a value other than zero to exaggerate differences, which is a common technique to convey a specific message or bias.

💡Statistical Significance

Statistical significance is a measure that indicates how likely it is that the observed results occurred by chance. The video clarifies that statistical significance does not equate to importance or usefulness, but rather it indicates that the results are reliable and not likely due to random chance. It is often misunderstood and misused to imply more than it actually does.

💡P-values

P-values are used in statistics to determine the strength of the evidence against a null hypothesis. A low p-value (typically ≤ 0.05) indicates strong evidence against the null hypothesis. The video discusses the common misunderstandings around p-values and how they are often misinterpreted as measures of certainty or importance, rather than as indicators of the likelihood of observing the data under the null hypothesis.

💡Data Dredging

Data dredging, also known as data fishing or p-hacking, is the practice of testing a large number of hypotheses or variables in a dataset until a statistically significant result is found. The video warns against this practice as it can lead to false conclusions because the same data is used to both find and validate a pattern, which increases the likelihood of a type I error (a false positive).

💡Bias in Media Reporting

Bias in media reporting refers to the tendency to present information in a way that promotes a particular point of view or agenda. The video provides several examples of how statistics can be manipulated or presented in a misleading way in news articles or political campaigns to sway public opinion or support a particular narrative.

Highlights

The speaker humorously introduces the topic of statistics by discussing the 'lies, damned lies, and statistics' adage.

The importance of understanding statistics in everyday life is emphasized, even without advanced degrees in math.

The concept of correlation is introduced, explaining its informal definition and its role in statistical analysis.

The difference between correlation and causation is clarified using the example of ice cream sales and drowning incidents.

The idea of a lurking variable, which can affect the perceived correlation between two variables, is discussed.

The speaker presents examples of spurious correlations, such as the number of movies with Nicolas Cage and drowning incidents.

The Simpsons paradox is introduced, demonstrating how data can be misinterpreted when analyzed in different ways.

Sampling bias is explained using the example of a survey that led to the incorrect headline 'Dewey Defeats Truman'.

The potential for bias in data visualization is highlighted, showing how charts can be manipulated to present a misleading narrative.

The concept of statistical significance is discussed, emphasizing that it does not equate to the importance of results.

P-values are explained as a measure of the probability of observing results under the null hypothesis, not a measure of certainty.

Data dredging, or the practice of searching for patterns in data until something significant is found, is critiqued for its potential to mislead.

The speaker advises on critical thinking when interpreting statistical data, questioning the context and funding behind studies.

The importance of not blindly trusting media headlines and seeking a deeper understanding of the data presented is stressed.

The speaker invites questions from the audience, promoting an interactive discussion on the topic of statistics and their misuse.

A question from the audience about the downside of data dredging is addressed, emphasizing the importance of not validating hypotheses on the same data used to find patterns.

The philosophical nature of discerning truth from lies in statistics is touched upon, suggesting that a deeper conversation is needed.

The speaker concludes by encouraging the audience to be vigilant and question the data they encounter, promoting a healthy skepticism.

Transcripts

Browse More Related Video

How to defend yourself against misleading statistics in the news | Sanne Blauw | TEDxMaastricht

How to lie with statistics

Lies, Damned Lies, and Statistics: The misapplication of statistics in everyday life.

The Antidote to Lies, Damned Lies and Statistics | Emily Bird | TEDxFremantle

Statistician Answers Stats Questions From Twitter | Tech Support | WIRED

Elementary Statistics Chapter 1

Related Tags

Statistics Misuse Data Interpretation Correlation vs Causation Sampling Bias Data Visualization Simpson's Paradox Statistical Significance P-Values Data Dredge Misleading Graphs Critical Thinking Python Community Data Analysis