Fixed Effect vs. Random Effects Models - Common Mistakes in Meta-Analysis and How To Avoid Them

In this introductory paragraph, the speaker, Michael Bornstein, sets the stage for a lecture on meta-analysis, focusing on the choice between fixed effect and random effects models. He clarifies that these models are often misunderstood and emphasizes the importance of selecting the correct model based on the research goals. Bornstein also mentions common mistakes made in meta-analysis and introduces his credentials and resources, including websites for software and workshops related to meta-analysis.

This paragraph delves into the fixed effect model, which is suitable for scenarios where all studies estimate the same underlying value. The speaker uses an example of estimating the mean math test score of seniors from a specific high school to illustrate the concept. The fixed effect model assumes that all studies are essentially identical, drawing samples from the same population and following the same protocol. The paragraph explains that the model is called 'fixed' because it deals with a specific, non-randomly selected population.

The speaker contrasts the fixed effect model with the random effects model, which is used when studies are drawn from a larger universe of populations. Using a variant of the previous example, the speaker describes how the random effects model applies when estimating math test scores across different schools in Brooklyn. This model assumes that each school has a unique mean score, and the studies represent a random sample from the universe of all schools. The paragraph highlights the differences in generalizability and the goals of the analysis between the two models.

This paragraph compares the two models on several aspects, including the sampling frame, generalizability, the meaning of the model names, and the goals of the analysis. The speaker explains that the fixed effect model is used when generalizing to a specific population and protocol, while the random effects model allows for generalization to an entire universe of studies. The paragraph also clarifies misconceptions about the models, such as the belief that they should yield similar results under certain conditions.

The speaker discusses the influence of large studies on the fixed and random effects models. Using a fictional example with five studies, the paragraph demonstrates how large studies with more precise estimates can disproportionately affect the results, particularly under the fixed effect model. The speaker explains the concept of weighting in meta-analysis and how it differs between the two models, with large studies receiving more weight under the fixed effect model and less under the random effects model.

In this paragraph, the speaker addresses common errors in selecting the appropriate model for meta-analysis. He criticizes the practice of using a heterogeneity test to decide between fixed and random effects models, arguing that the choice should be based on the understanding of the sampling frame. The speaker provides examples from published papers where this mistake has been made, leading to incorrect model selection and subsequent analysis.

The speaker dispels several myths regarding fixed and random effects models. He clarifies that the random effects model does not always yield a higher effect size or a more conservative estimate than the fixed effect model. The paragraph emphasizes that the choice between models should be based on the nature of the data and the research question, rather than arbitrary criteria or misconceptions.

In the final paragraph, the speaker discusses the limitations of the random effects model, particularly when applied to studies pulled from the literature. He points out the difficulty in defining the universe of effect sizes and the challenges of obtaining a random sample from this universe. The speaker also highlights the importance of having a precise estimate for the between-study variance, noting that with a small number of studies, this estimate may not be reliable. He concludes by acknowledging the value of the random effects model while urging researchers to be aware of its potential pitfalls.