Statistical Significance and p-Values Explained Intuitively

Data Demystified
22 Oct 202008:56
EducationalLearning
32 Likes 10 Comments

TLDRIn this video, Jeff Gallick from 'Date Demystified' explains the concept of statistical significance in scientific studies. He clarifies that while we cannot prove something to be true, we can provide evidence against a null hypothesis, which is the assumption of no effect or no difference. The video delves into the meaning of 'statistically significant results', typically indicated by a p-value less than 0.05. This p-value represents the probability of observing the data (or more extreme) if the null hypothesis were true. A p-value below 0.05 suggests that the observed results are unlikely if the null hypothesis holds, thus providing enough evidence to reject it. Gallick also discusses the arbitrary nature of the 0.05 threshold, which is a convention established by Ronald Fisher in the early 20th century. He suggests that while some argue for the abandonment of p-value significance testing in favor of confidence intervals and Bayesian methods, the convention aids in consistent scientific communication.

Takeaways
  • 🧐 **Statistical Significance**: In science, we can't prove something to be true; instead, we provide evidence against a null hypothesis, which is a statement of no effect or no difference.
  • 📉 **P-Value**: A p-value is a statistic that measures the strength of the evidence against the null hypothesis. A smaller p-value indicates stronger evidence to reject the null hypothesis.
  • 🔍 **Threshold of 0.05**: The convention of considering a p-value less than 0.05 as statistically significant was chosen arbitrarily and is not an absolute measure of truth.
  • 🔄 **Repeatability**: Repeated experiments that show a consistent result increase our confidence in rejecting the null hypothesis, though we never achieve absolute certainty.
  • 🌐 **Data-Rich World**: The video aims to equip viewers with the knowledge to thrive in a world where data and statistical analysis play a crucial role.
  • 📚 **Null Hypothesis**: In statistics, the null hypothesis is a default position that assumes no effect or no relationship between variables, which is what we test against.
  • 🔎 **Evidence Against Null**: When we observe a result that strongly suggests the null hypothesis is false, we gather evidence to reject it, not prove it wrong.
  • 💊 **Example of Drug Efficacy**: The video uses the example of testing a drug's effectiveness against heart disease to illustrate the process of statistical significance testing.
  • 📊 **Confidence and P-Value**: A lower p-value (e.g., 0.01) means that if the null hypothesis were true, the observed result would be very unlikely, thus increasing confidence in rejecting the null hypothesis.
  • 🤔 **Arbitrary Nature of 0.05**: The choice of 0.05 as a threshold for statistical significance is a historical convention without a scientific basis for being the ideal cutoff.
  • ⚖️ **Alternatives to P-Values**: Some argue for focusing on confidence intervals, Bayesian estimates, and effect sizes as alternatives to traditional p-value significance testing for a more nuanced understanding of evidence.
Q & A

  • What is the main idea behind statistical significance testing in science?

    -The main idea behind statistical significance testing is that we can never prove something to be true; instead, we provide evidence against a null hypothesis, which is an assumption of no effect or no difference.

  • What does the term 'statistically significant result' imply?

    -A 'statistically significant result' implies that there is enough evidence to reject the null hypothesis, suggesting that the observed effect or difference is unlikely to have occurred by chance alone.

  • What is the role of the null hypothesis in statistical testing?

    -The null hypothesis serves as a default assumption of no effect or no relationship between variables. It is what we assume to be true until evidence from data analysis suggests otherwise.

  • Why can't we prove something to be true in science?

    -We can't prove something to be true in science because it would require observing every possible condition across all of time, which is impossible. Instead, we can only provide evidence against a claim or assumption.

  • What is the significance of the p-value in determining statistical significance?

    -The p-value indicates the probability of observing the data (or more extreme data) if the null hypothesis were true. A smaller p-value suggests stronger evidence against the null hypothesis, with a common threshold for significance being p < 0.05.

  • Why is the threshold for statistical significance often set at 0.05?

    -The threshold of 0.05 is arbitrary and was chosen by convention, likely by Ronald Fisher in the early 20th century. It is used as a standard to determine if the evidence is strong enough to reject the null hypothesis.

  • What is the difference between proving a hypothesis and rejecting a null hypothesis?

    -Proving a hypothesis would require demonstrating its truth in every possible scenario, which is not feasible. Rejecting a null hypothesis, on the other hand, involves showing that the observed data is unlikely under the assumption of the null hypothesis, providing evidence against it.

  • How does the concept of 'innocent until proven guilty' relate to the null hypothesis in statistics?

    -The concept of 'innocent until proven guilty' is analogous to the null hypothesis in that it represents the default position that is assumed to be true unless sufficient evidence is presented to the contrary.

  • What are some criticisms of using statistical significance testing with a p-value threshold?

    -Critics argue that the reliance on a p-value threshold can be misleading, as it does not account for the size of the effect, the power of the study, or the potential for false positives. Some suggest focusing on confidence intervals, Bayesian methods, or effect sizes instead.

  • What is the meaning of a p-value of 0.01 in the context of a study?

    -A p-value of 0.01 means that if the null hypothesis were true, there is only a 1% chance of observing a result as extreme as the one obtained in the study, suggesting strong evidence against the null hypothesis.

  • Why might someone choose a different p-value threshold, such as 0.01 or 0.15?

    -A researcher might choose a different p-value threshold to adjust the stringency of the evidence required to reject the null hypothesis. A lower threshold like 0.01 indicates a higher standard for evidence, while a higher threshold like 0.15 indicates a more lenient standard.

  • What are some alternatives to traditional statistical significance testing that are sometimes recommended?

    -Some alternatives to traditional significance testing include confidence intervals, which provide a range of values within which the true effect is likely to fall; Bayesian estimates, which incorporate prior knowledge and update it with new data; and effect sizes, which measure the magnitude of the difference or strength of the relationship between variables.

Outlines
00:00
📊 Understanding Statistical Significance

This paragraph introduces the concept of statistical significance in scientific studies. The host, Jeff Gallick, explains that while we can't prove something to be true, we can provide evidence against a null hypothesis. He uses the example of a ball falling to the ground to illustrate how repeated observations can build evidence against a hypothesis. The paragraph also touches on the idea that statistical significance is about comparing results to a null hypothesis and the role of p-values in determining if results are statistically significant.

05:02
🔍 The Role of P-Values in Statistical Significance

In this paragraph, Jeff discusses the role of p-values in statistical testing. He explains that a p-value represents the probability of observing a given result if the null hypothesis were true. A smaller p-value indicates stronger evidence against the null hypothesis. He uses the example of a drug trial to illustrate how different p-values can lead to different conclusions about the drug's effectiveness. Jeff also explains that a p-value of less than 0.05 is traditionally considered statistically significant, meaning there's less than a 5% chance of observing the results if the null hypothesis were true. He notes that the choice of 0.05 as a cutoff is arbitrary and that different disciplines might use different thresholds. Finally, he mentions that while statistical significance testing is a common approach, some argue for focusing on confidence intervals, Bayesian estimates, and effect sizes instead.

Mindmap
Keywords
💡Statistical Significance
Statistical significance refers to a result that is unlikely to have occurred by chance alone. In the context of the video, it is used to determine if there is enough evidence to reject a null hypothesis. The video explains that a statistically significant result typically has a p-value less than 0.05, which means that if the null hypothesis were true, there is less than a 5% chance of observing the results obtained from the study.
💡P-Value
A p-value is a statistic that measures the strength of the evidence against the null hypothesis. It is used to decide whether to reject the null hypothesis in a statistical test. In the video, it is mentioned that a smaller p-value indicates greater confidence in rejecting the null hypothesis. For example, a p-value of 0.01 suggests that there is a 1% chance of observing the results if the null hypothesis were true, thus providing strong evidence against it.
💡Null Hypothesis
The null hypothesis is a default position that assumes there is no effect or no relationship between variables being studied. It is what researchers assume to be true until evidence suggests otherwise. In the video, the null hypothesis is likened to the presumption of innocence in a legal trial. It is used as a benchmark to compare the observed data against, and if the observed data is unlikely under the null hypothesis, it may be rejected in favor of an alternative hypothesis.
💡Data-Rich World
The term 'data-rich world' refers to the modern era where vast amounts of data are available for analysis. The video's host, Jeff, mentions this to emphasize the importance of understanding statistical concepts like statistical significance in a world filled with data and information. It underscores the need for individuals to be equipped with the knowledge to interpret and make decisions based on data.
💡Evidence
In the context of the video, evidence is used to refer to the results or data collected from an experiment or study that either supports or contradicts a hypothesis. The video explains that in science, one does not prove something to be true but rather provides evidence against a claim. For instance, repeatedly observing a ball fall to the ground when dropped provides evidence against the claim that a dropped ball will not fall.
💡Confidence Intervals
Confidence intervals are a range of values, derived from a data set, that likely contains the value of an unknown parameter. They are used to indicate the reliability of an estimate. The video suggests that some researchers prefer to focus on confidence intervals rather than p-values as a measure of statistical significance. They provide a range within which the true value is likely to fall with a certain level of confidence.
💡Bayesian Estimates
Bayesian estimates are a type of statistical inference where the evidence from data is used to update the probabilities of hypotheses. Unlike traditional statistical significance testing, Bayesian methods incorporate prior beliefs or existing knowledge into the analysis. The video briefly mentions Bayesian estimates as an alternative approach to traditional significance testing, suggesting a more nuanced understanding of evidence and belief updating.
💡Effect Size
Effect size is a measure of the magnitude of a phenomenon or the strength of the relationship between variables. It is distinct from statistical significance, which only tells us if the result is unlikely to be due to chance. The video points out that focusing on effect size can provide a more meaningful understanding of the practical significance of research findings, rather than just their statistical significance.
💡Randomized Experiment
A randomized experiment is a study in which participants are assigned to different groups by chance, often to control for confounding variables and to assess the causal effect of an intervention. The video mentions randomized experiments in the context of testing the effectiveness of a drug, emphasizing the importance of this methodological approach in providing strong evidence for rejecting or failing to reject the null hypothesis.
💡False Positive Rate
The false positive rate, or Type I error rate, is the probability of incorrectly rejecting a true null hypothesis. It is related to the p-value and the significance level chosen for a statistical test. The video touches on this concept when discussing the implications of setting a threshold for statistical significance, noting that a lower threshold reduces the risk of false positives but may also increase the risk of false negatives.
💡Ronald Fisher
Ronald Fisher was a British statistician who made significant contributions to the development of modern statistical methods. The video credits him as the person who established the 0.05 significance level as a convention in statistical testing. His work has had a profound impact on how scientific research is conducted and evaluated, particularly in setting standards for determining statistical significance.
Highlights

Jeff Galick, the host of Date Demystified, aims to equip viewers with the information needed to thrive in a data-rich world.

The concept of 'statistical significance' is often mentioned in scientific studies and refers to a p-value less than 0.05.

Statistical significance testing is about providing evidence against a null hypothesis, not proving it true.

Science provides evidence against claims rather than absolute proof, due to the impossibility of observing every condition across time.

The null hypothesis is an assumption of innocence until proven guilty, which is what we assume to be true until evidence suggests otherwise.

Statistical significance is determined by comparing the result of an analysis to a null hypothesis, similar to a jury convicting a defendant.

The p-value represents the probability of observing the data (or more extreme) if the null hypothesis were true.

A p-value less than 0.05 indicates that there is strong enough evidence to reject the null hypothesis.

The threshold of 0.05 for statistical significance is arbitrary and was chosen by convention, likely by Ronald Fisher in the early 20th century.

Choosing a smaller p-value cutoff like 0.01 provides stronger evidence to reject the null hypothesis.

A higher p-value, such as 0.60, suggests that the observed results are likely even if the null hypothesis is true, indicating weak evidence.

The p-value of 0.01 means that if the null hypothesis were true, there's only a 1% chance of observing the data or more extreme results.

The video discusses the importance of shared scientific language for effective communication and knowledge growth.

Some argue that statistical significance testing should be abandoned in favor of confidence intervals, Bayesian estimates, and effect sizes.

The video simplifies complex statistical concepts to provide an intuitive understanding of significance testing.

The host encourages viewers to engage by liking, subscribing, and turning on notifications for new content.

The video ends with a call to action for viewers to comment if they are interested in learning more about specific statistical topics.

Transcripts

Browse More Related Video

P Value and Hypothesis Testing Simplified|P-value and Hypothesis testing concepts in Statistics

p-values: What they are and how to interpret them

p-value - easily explained with an example

How P-Values Help Us Test Hypotheses: Crash Course Statistics #21

P-values and significance tests | AP Statistics | Khan Academy

What Is A P-Value? - Clearly Explained

Rate This

5.0 / 5 (0 votes)

Thanks for rating:

Related Tags
Statistical SignificanceP-ValuesNull HypothesisData AnalysisScience CommunicationEvidence BasedScientific MethodResearch MethodsData InterpretationStatistical Threshold