Statistical vs Practical Significance Compared

Data Demystified
26 Nov 202008:28
EducationalLearning
32 Likes 10 Comments

TLDRIn this insightful video, Jeff Gallick from 'Data Demystified' challenges the conventional reliance on statistical significance as the sole determinant of a study's validity. He introduces the concept of 'meaningful significance,' emphasizing that while statistical significance establishes whether an observed effect is likely due to chance, it doesn't necessarily convey the practical importance of the result. Gallick illustrates this with examples from medicine, economics, and education, where interventions might show statistically significant benefits but may not be worth the investment if the impact is minimal. He encourages viewers to consider both statistical and meaningful significance when evaluating the value of a study, policy, or intervention. The video is a call to action for a more nuanced understanding of data, advocating for critical thinking that goes beyond statistical thresholds to assess the true impact and practical implications of findings.

Takeaways
  • 📊 **Statistical Significance:** It's a common measure used to determine if an observed result is likely due to chance or a real effect.
  • 💊 **Effectiveness vs. Significance:** A new drug or policy needs to show statistical significance to be considered viable, but that doesn't always mean it has meaningful impact.
  • 🔍 **Meaningful Significance:** Just because a result is statistically significant doesn't mean it's practically important or worth the investment.
  • 🤔 **Necessary but Not Sufficient:** Statistical significance is necessary to rule out chance, but it's not enough to justify implementing an intervention.
  • 📈 **Effect Size:** This measures the magnitude of an effect and is crucial in determining how impactful an intervention is.
  • 💰 **Cost-Benefit Analysis:** Consider the costs versus benefits when evaluating the meaningfulness of an intervention, such as a vaccine or policy change.
  • 📉 **Small but Significant:** An intervention can be statistically significant with a small effect size, which may not justify its implementation.
  • ⚖️ **Trade-offs:** Decision-makers should weigh the statistical significance against the practical implications and costs.
  • 🧮 **Educational Example:** In education, a small improvement in test scores might not justify a large-scale change in teaching methods.
  • 🤝 **Policy Decisions:** Policy interventions should not only be statistically proven but also have a meaningful effect size to be considered successful.
  • ✅ **Critical Evaluation:** When assessing results, ask if they are statistically significant and if they are meaningful in a real-world context.
Q & A

  • What is considered the gold standard in various disciplines to show that something is true?

    -Statistical significance is often considered the gold standard in a variety of disciplines to demonstrate that something is true.

  • What is the primary purpose of statistical significance in the context of a new drug development?

    -Statistical significance is used to show that a new drug is effective enough to be considered viable.

  • What is the difference between statistical significance and meaningful significance?

    -Statistical significance determines if a result is likely due to chance or not, while meaningful significance assesses the practical importance or impact of that result.

  • Why might a statistically significant result not be worth implementing?

    -A statistically significant result might not be worth implementing if the effect size is very small and the costs or resources required to implement it are high.

  • What is an effect size and why is it important?

    -Effect size is a measure that indicates the magnitude of an intervention's impact. It is important because it tells us how impactful an intervention is, not just whether it works.

  • How can effect sizes be used in decision-making?

    -Effect sizes can be used to make trade-offs, such as deciding whether to proceed with the distribution of a vaccine, invest in a policy intervention, or change educational strategies.

  • What is the necessary but not sufficient condition for a result to be considered statistically significant?

    -The necessary but not sufficient condition for statistical significance is that the observed difference or relationship between variables is not likely due to chance.

  • Why is it important to consider both statistical and meaningful significance when evaluating a policy decision?

    -Considering both types of significance is important because it helps to determine not just whether a policy will work, but also how effective it is and whether the benefits outweigh the costs.

  • How might a small but statistically significant increase in wages due to free college tuition be evaluated?

    -The increase would be evaluated based on whether the cost of implementing free college tuition is justified by the magnitude of the wage increase, even if it is statistically significant.

  • null

    -null

  • What is the role of statistical significance in the context of a medical vaccine example?

    -Statistical significance in a medical vaccine example helps to confirm that the vaccine's effectiveness in reducing infection rates is not due to chance, but it does not inform about the magnitude of the vaccine's impact.

  • How can one intuitively compare different types of interventions using effect sizes?

    -One can compare different interventions by looking at the effect sizes in a common metric, such as percentage terms, which allows for a direct comparison of the magnitude of impact across different interventions.

  • What are common language effect sizes and why are they useful?

    -Common language effect sizes are a way to express the magnitude of an effect in terms that are easily understood by non-experts. They are useful for communicating the practical significance of results to a broader audience.

  • Why might a statistically significant result with a tiny difference across groups still be important?

    -A statistically significant result with a tiny difference can be important if the context or the condition being measured is such that even a small change has significant practical implications or if the result is part of a cumulative effect over time.

Outlines
00:00
📊 Understanding Statistical Significance vs. Meaningful Significance

The first paragraph discusses the concept of statistical significance, which is often used as a benchmark to validate findings in various fields such as pharmaceuticals, policy interventions, and marketing. However, it emphasizes that statistical significance does not necessarily equate to meaningful significance. The speaker, Jeff Gallick, introduces himself and his mission to help viewers navigate the data-rich world. He proposes an intuitive framework for evaluating results based not just on their statistical validity but also on their practical importance or 'effect size,' which measures the magnitude of an intervention's impact. The paragraph also uses the example of a vaccine to illustrate the difference between statistical and meaningful significance, highlighting that even a small reduction in disease risk can be statistically significant but may not be substantial enough to justify the costs associated with the vaccine.

05:00
📈 The Importance of Considering Effect Size in Decision Making

The second paragraph expands on the idea that while statistical significance is crucial to establish that a result is not due to chance, it is equally important to consider the effect size to determine the practical implications of an intervention. It uses examples from economics and education to illustrate how a statistically significant result might not translate into a meaningful impact if the effect size is small. The speaker suggests a framework for critical assessment that involves two key questions: whether the result is statistically significant and whether it is meaningful. The paragraph concludes by encouraging viewers to look beyond statistical significance to make informed decisions about the implementation of policies or interventions. It also invites viewers to engage with the content by liking, subscribing, and enabling notifications for new content.

Mindmap
Keywords
💡Statistical Significance
Statistical significance is a measure used in data analysis to determine whether an observed difference or result is likely due to chance or not. It is a critical concept in scientific research, often used to establish that a result is not merely a random occurrence. In the video, it is discussed as a minimum requirement to prove that a new drug, policy, or advertisement is effective. However, the video emphasizes that statistical significance does not necessarily equate to meaningful significance, which is a key message of the video.
💡Meaningful Significance
Meaningful significance refers to the practical importance or impact of a result, beyond just being statistically significant. It asks not just whether something is true, but also whether it matters enough to be considered valuable or worth implementing. In the context of the video, meaningful significance is contrasted with statistical significance to highlight that a result might be statistically proven but still may not be substantial enough to warrant action or investment.
💡Effect Size
Effect size is a measure of the magnitude of a result, indicating how large the difference is between groups or the strength of the relationship between variables. It is used to quantify the importance of a finding, beyond just its statistical significance. The video uses effect size to illustrate how impactful an intervention, such as a vaccine or a policy change, is. For example, a vaccine that reduces disease by 10% may have a different effect size and thus a different level of impact compared to one that reduces disease by 20%.
💡Data Analysis
Data analysis is the process of examining and interpreting data to draw meaningful conclusions. It is central to the video's discussion as it is through data analysis that statistical and meaningful significance are determined. The video script discusses how data analysis can be used to evaluate the efficacy of a vaccine, the effectiveness of a policy intervention, or the success of an advertising strategy.
💡Policy Intervention
A policy intervention refers to a strategic action or initiative implemented by a government or organization to address a particular issue or to achieve a specific outcome. In the video, the concept is used to discuss how statistical significance alone may not be sufficient to justify the costs and efforts associated with implementing a new policy, such as providing free college education. The meaningful impact of such an intervention is also considered.
💡Vaccine Efficacy
Vaccine efficacy is the extent to which a vaccine is effective in preventing a disease. It is a key example used in the video to illustrate the difference between statistical significance and meaningful significance. A vaccine may show statistical significance in reducing disease spread, but the video argues that its meaningful significance, or the extent of its efficacy, is also crucial in deciding whether to invest in its production and distribution.
💡Risk Reduction
Risk reduction refers to the decrease in the probability of an adverse event occurring, such as the risk of disease. The video discusses how a vaccine might statistically significantly reduce the risk of disease, but the actual amount of risk reduction (e.g., 1% vs. 100%) is essential in determining the vaccine's meaningful significance and whether it is worth the investment.
💡Manufacturing and Distribution
Manufacturing and distribution pertain to the processes of producing a product, such as a vaccine, and delivering it to the end-users. In the context of the video, these processes are mentioned in relation to the costs and logistics involved in making a vaccine available to the public. The video suggests that the costs associated with manufacturing and distribution should be weighed against the meaningful significance of the vaccine's efficacy.
💡Research and Development (R&D)
Research and development (R&D) involves creating new knowledge or improving existing knowledge through scientific research and technological innovation. The video script suggests that if a current vaccine's effect is statistically significant but has a small effect size, resources might be better spent on R&D to find a more effective vaccine rather than investing in the current one's distribution.
💡Educational Intervention
An educational intervention is a strategy or program designed to improve educational outcomes. The video uses the example of changing the way math is taught in primary school to demonstrate how a statistically significant result (a 1% increase in proficiency) might not be meaningful enough to justify the costs of retraining teachers.
💡Trade-offs
Trade-offs are decisions where one must choose between two options, both of which have positive and negative aspects. The video discusses the importance of considering trade-offs when evaluating the implementation of interventions. It suggests that even if an intervention is statistically significant, one must weigh the costs and benefits to determine if the intervention's meaningful significance justifies its implementation.
Highlights

Statistical significance is often considered the gold standard in various disciplines to show that something is true.

Statistical significance is a minimum requirement to show the efficacy of a new drug, the effectiveness of a policy intervention, or the choice between different advertisements.

Statistical significance fails to consider meaningful significance, which is the importance of a result beyond just being true.

The speaker, Jeff Gallick, aims to provide an intuitive framework for evaluating the meaningfulness of a result.

Statistical significance determines if an observed difference is likely due to chance or something else.

Having enough data and confidence allows one to claim statistical significance, suggesting the results aren't due to luck.

Effect size measures the impact of an intervention, not just its effectiveness.

Effect sizes are crucial for making trade-offs, such as whether to distribute a vaccine or invest in further research.

Statistical significance can be present even with a very small effect size, such as a 1% reduction in infection rates.

Meaningful significance asks if a statistically significant result is worth the investment or action.

In economics, the cost versus benefit analysis is crucial, and a statistically significant result must also be meaningful to be worth implementing.

Policy interventions should be evaluated not just on their statistical significance but also on their effect size and practical impact.

The framework for assessing statistical results involves asking if the result is statistically significant and if it is meaningful.

Critical evaluation of any policy decision should consider both the effectiveness and the extent of its impact.

Statistical significance can lead to overlooking the practical significance and necessary trade-offs of an intervention.

The video encourages viewers to think beyond statistical significance to judge whether a result is truly meaningful.

The speaker acknowledges that some topics, like computing effect sizes and comparisons between interventions, have been glossed over for the sake of focusing on the core intuition.

The video concludes by encouraging viewers to engage with the content, subscribe to the channel, and stay updated with new content.

Transcripts

Browse More Related Video

What do my results mean Effect size is not the same as statistical significance. With Tom Reader.

Statistical Significance and p-Values Explained Intuitively

Understanding Statistical Significance - Statistics help

What is Effect Size? Statistical significance vs. Practical significance

P value versus effect size

Effect size in meta analysis

Rate This

5.0 / 5 (0 votes)

Thanks for rating:

Related Tags
Statistical SignificanceMeaningful ImpactData AnalysisDecision MakingEffect SizePolicy InterventionHealthcare EconomicsEducation ReformVaccine EfficacyResearch MethodsData InterpretationJeff GalickData Demystified