The Rules of the Game Called Psychological Science

It has long been argued that the combined outcomes of NHST in the scientific literature are too good to be true (Fanelli, 2010; Fiedler, 2011; Ioannidis, 2008a; Sterling, 1959; Vul, Harris, Winkielman, & Pashler, 2009). Fanelli (2010) documented that over 80% of scientific publications in various sciences report positive results and that the psychological literature shows the highest prevalence of positive outcomes. Sterling (1959) and Sterling, Rosenbaum, and Weinkam (1995) showed that in 97% (in 1958) and 96% (in 1986–1987) of psychological studies involving the use of NHST, H0 was rejected at α = .05. Although it should be noted that psychological papers report a host of test results (Maxwell, 2004), the abundance of positive outcomes is striking because effect sizes (ESs) in psychology are typically not large enough to be detected by the relatively small samples used in most studies (i.e., studies are often underpowered; Cohen, 1990).

The power of statistical tests depends on the nominal significance level (typically .05), the sample size, and the underlying ES, such as Cohen’s d for between-group mean comparisons. According to Marszalek, Barber, Kohlhart, and Holmes (2011), the median total sample size in four representative psychological journals (Journal of Abnormal Psychology, Journal of Applied Psychology, Journal of Experimental Psychology: Human Perception and Performance, and Developmental Psychology) was 40. This finding is corroborated by Wetzels et al. (2011), who found a median cell size of 24 in both between- and within-subjects designs in their large sample of t tests from Psychonomic Bulletin & Review and Journal of Experimental Psychology: Learning, Memory and Cognition. The average ES found in meta-analyses in psychology is around d = 0.50 (Anderson, Lindsay, & Bushman, 1999; Hall, 1998; Lipsey & Wilson, 1993; Meyer et al., 2001; Richard, Bond, & Stokes-Zoota, 2003; Tett, Meyer, & Roese, 1994), which might be an overestimation of the typical ES given the biases we discuss below. Nevertheless, the typical power in our field will average around 0.35 in a two independent samples comparison, if we assume an ES of d = 0.50 and a total sample size of 40. This low power in common psychological research raises the possibility of a file drawer (Rosenthal, 1979) containing studies with negative or inconclusive results. Publication bias can have dire consequences, as illustrated recently by clear failures to replicate medical findings (Begley & Ellis, 2012; Prinz, Schlange & Asadullah, 2011). On the basis of surveys of researchers and a study of the fate of studies approved by the institutional review board of a major U.S. university, the percentage of unpublished studies in psychology is estimated to be at least 50% (Cooper, DeNeve, & Charlton, 1997; Coursol & Wagner, 1986; Shadish, Doherty, & Montgomery, 1989), but the problem goes beyond widespread failure to publish. Statistical textbooks advise the use of formal a priori power estimates, but in a recent sample of psychological papers with NHST (Bakker & Wicherts, 2011), only 11% referred to power as a rationale for the choice of sample size or design. Although power estimates can also be done informally, the typical study in psychology appears to be underpowered. If a study’s power equals 0.50, the chance to find a significant result equals that of correctly predicting “heads” in a coin flip. The number of “heads” presented in the psychological literature (and in other literatures) suggests a problem. Although one author has explained this by claiming that researchers are psychic (Bones, 2012), we think that they just act strategically.

The common lack of power is well illustrated by studies of the (positive) association between infants’ habituation to a given stimulus and their later cognitive ability (IQ). One often-cited meta-analysis (McCall & Carriger, 1993) collated 12 studies of the correlation between measures of habituation during children’s first year of life and IQ as measured between 1 and 8 years of age. In the funnel plot (Light & Pillemer, 1984) of Figure 1, these 12 Fisher-transformed (normalized) correlations are plotted against the inverse of the standard error (SE) in each study. The SE depends on sample size and equals $\sqrt{1/N-3}$. The white area represents the 95% confidence interval (or area) under H₀ = 0, and study outcomes that fall in the grey area are significant at α = .05 (two-tailed). The straight dotted line represents the estimated underlying ES from a fixed effect meta-analysis (Z_r = .41, which corresponds to r = .39 and d = .85) and the curved dotted lines represent the 95% confidence interval around that estimate of the alternative hypothesis (H_A; study outcomes invariably fell in this interval and so they appear to be homogeneous; Q = 6.74, DF = 11, p = .820). The upward narrowing of both 95% confidence intervals in the funnel reflects larger power to reject H₀ in large samples. On the right hand side, we depicted the power to reject H₀ given the sample size (SE level) as implied by the estimated underlying ES of .41. For instance, on the top of Figure 1, we depicted the distributions under H₀ and H_A for N = 50 (1/SE = 6.86), which corresponds to a power of .807. As we go down the funnel, SEs become larger and so outcomes should deviate more strongly from the estimate of H_A. It is noteworthy that (a) all but three of the studies have a power below .80, (b) the correlation differed significantly from zero in all but one study, and (c) the study outcomes are clearly not evenly distributed in the right- and left-hand side of the funnel associated with H_A. In fact, the two largest studies showed the weakest link between infant cognition and later IQ, whereas the smaller studies all lie on the right-hand side the funnel (i.e., in the grey area where p < .05). Such funnel plot asymmetry is awkward and can be tested (Sterne & Egger, 2005) by regressing outcomes on sample sizes (or SEs) across the 12 studies: Z = 2.24, p = .025. The median sample size of these studies was 25 and their typical (median) power equaled .488. Under the assumption that studies are independent, the expected number of significant findings on the basis of this power analysis (i.e., the sum of power values) is 6.71, and so a positive outcome in 11 out of such 12 underpowered studies is unlikely. Ioannidis and Trikalinos (2007) proposed a straightforward χ² test for such an excess of significant findings: χ² (DF = 1) = 6.21, p = .013. So this meta-analysis shows the typical signs of publication bias and results that are too good to be true (Francis, 2012b). One possible explanation is that studies with nonsignificant or lower correlations were missing from the meta-analysis. In addition, research with infants is not easy and seldom are these analyses carved in stone. Statistical choices concerning exclusion of data points, outliers, and operationalization of dependent variables require choices that are often arbitrary and so provide researchers degrees of freedom (Simmons et al., 2011) in their analyses.

The excess of significant findings may partly be explained by researchers’ exploitation of these degrees of freedom in their pursuit of significant outcomes (Fiedler, 2011; Wicherts, Bakker, & Molenaar, 2011) and by the fact that it is easier to find a significant effect in multiple small studies rather than one larger study. The use of multiple small studies rather than a larger one gives the researcher the opportunity to make small alterations to the research design and provides ample opportunity for capitalizing on chance.

Simmons et al. (2011) illustrated how easy it is to inflate Type I error rates when researchers employ hidden degrees of freedom in their analyses and design of studies (e.g., selecting the most desirable outcomes, letting the sample size depend on results of significance tests). John, Loewenstein, and Prelec (2012) surveyed over 2,000 psychological researchers and found a majority of them to admit to use at least some of these questionable research practices (QRPs). For instance, the majority admitted to having ever failed to report all of the dependent measures in a study. Forty-eight percent admitted to having only report studies that “worked” (which we take to imply p < .05), whereas 57% acknowledged to having used sequential testing (cf. Wagenmakers, 2007) in their work. Such practices lead to inflated ESs and increased false positive rates (Ioannidis, 2005, 2008b; Simmons et al., 2011).

Suppose psychology were a game in which players have to gather a significant result in a particular direction. Players have resources to gather data from N participants and can choose between these options:

Strategies 3 and 4 imply publication bias in the traditional sense, whereas Strategies 2 and 4 relate to the analysis of the data. So what is the winning strategy? We simulated data (see the online Appendix at http://pps.sagepub.com/supplemental for details) on the basis of sample sizes and ESs that are typical for psychology and found a clear answer. The left panel of Figure 2 gives the proportion of researchers who gather at least one significant finding (p < .05) under these four strategies. Note that we simulated one-sided results (i.e., directional hypotheses) but employed two-sided tests, which should be significant in the expected direction. So the (combined) Type I error rate (i.e., when ES = 0) is .025 for the large study and 1 − (1 − .025)^5 = .119 for the five small studies.

The upshot of the simulation results is that when the true ES is small executing multiple small and underpowered studies represents the optimal strategy for individual players to generate a p value of less than .05. Furthermore, the use of QRPs pays off, especially with small samples and ESs. Besides, many players need not even perform all five studies—for example, with a cell size of 20 and an ES of 0.5 they need on average 2.58 studies (expected total N = 103) without QRPs and 1.14 studies (expected total N = 66) with QRPs. Combined with the selection of significant results through publication decisions, these strategies may explain why so many psychological researchers continue to run underpowered studies yet almost always report significant results.

Science is not a game. The optimal strategies described above lead to inflated Type I errors of up to 40% and inflate genuine effects. We calculated the bias in our simulation as the difference between the average estimated ES and the true ES. Results are presented in the right panel of Figure 2. With large samples (Strategy 1), there is no systematic bias. For large samples with QRPs (Strategy 2), the bias goes to zero with a larger true ES, which is to be expected because of the larger power under this scenario (QRPs are not required for “winning”). The bias is large with multiple small studies (Strategies 3 and 4). In the typical psychological study (cell size 20 and true ES = .5), the biases with and without QRPs are 0.327 and 0.158, respectively. With smaller cell sizes and an ES of .5, the bias can be as large as 0.913. When multiple small studies are combined with QRPs (Strategy 4), the bias is large for small true ES, but decreases with larger true ES, possibly because of the adding of subjects (in Step b). Even those who ignore p values of individual studies will find inflated ESs in the psychological literature if a sufficient number of researchers play strategically, which indeed many psychological researchers appear to do (John et al., 2012).

Our field lacks clear codes of conduct considering the use of these analytic strategies (Sterba, 2006) and many reviewers tend not to accept p values above .05 (Mahoney, 1977), possibly because this presents an easy heuristic (Hoekstra, Finch, Kiers, & Johnson, 2006; Nelson, Rosenthal, & Rosnow, 1986). Sole studies are seldom definitive but even knowledgeable researchers tend to underestimate the randomness associated with small samples (Tversky & Kahneman, 1971). At the end of the day it is all about replication.

A set of reasonably similar replications can be used to determine robustness of findings and to study signs of the use of the strategies described above. In our simulations, we used fixed effects and applied Sterne and Egger’s (2005) test of funnel plot asymmetry and Ioannidis and Trikalonis’ (2007) test for an excess of significant findings. Both methods are described above and functioned well in the simulation (see the online Appendix at http://pps.sagepub.com/supplemental and Fig. 4), although it is important to note that they are sensitive to actual heterogeneity of the underlying ESs (Ioannidis & Trikalinos, 2007; Sterne & Egger, 2005). For instance, funnel plot asymmetry may arise if smaller studies tap on stronger underlying effects because they are done in relatively more controlled settings. Therefore, these tests are best applied to relatively homogenous sets of studies as defined in the realm of meta-analysis.

To gather a representative sample of sets of psychological studies that concern the same phenomenon or at least highly similar phenomena, we retrieved from PsycARTICLES all 108 peer-reviewed articles published in 2011 that contained the strings “research synthesis,” “systematic review,” or “meta-anal*” in title and/or abstract. Subsequently, we randomly selected 11 useful meta-analyses (10% of the total). We only included meta-analyses that reported the ESs and standard errors (or sample sizes) of primary studies. From each meta-analysis, we retrieved the subset (as selected by the authors of the meta-analyses) of at least 10 primary studies that was the most homogenous subset in terms of Higgins’ I². We assumed that the meta-analysts employed rigorous inclusion and exclusion criteria and that they correctly determined ESs, and we feel confident that the primary studies in each of the fields are sufficiently comparable to be considered replications.

The selected (subsets from) meta-analyses are given in Table 1 together with tests for homogeneity, excess of significant findings, and funnel plot asymmetry. The average impact factor of the journals in which the meta-analyses appeared was 4 (see the online Appendix at http://pps.sagepub.com/supplemental for full references). Meta-analyses were from clinical, counseling, educational, evolutionary, developmental, family, and industrial/organizational psychology. The medians of the sample sizes align with those found in the wider literature, although the median ES (d = .37) was slightly lower than d = .50, as described earlier.

Figure 3 depicts the funnel plots of the 11 meta-analyses. Tests for funnel plot asymmetry (with an α of .10 as suggested by various authors; Ioannidis & Trikalinos, 2007) were significant in four instances (36%). In three instances, we found signs of an excess of significant results (27% at α = .10). These results replicate earlier indicators of the prevalence of funnel plot asymmetry in 99 psychological meta-analyses (Ferguson & Brannick, 2012; Levine, Asada, & Carpenter, 2009) and the finding of an excess of significant results in four areas of psychological research (Francis, 2012a, 2012b, in press).

To get a feel for the likelihood of biases in the actual meta-analyses, we simulated results for 16 meta-analyses with 100 studies each. These meta-analyses are presented in Figure 4 (see the online Appendix at http://pps.sagepub.com/supplemental for details), based on the four strategies as described above (Column 1 through 4) and four levels of true ES (d = 0.0, 0.2, 0.5, or 0.8; Rows A through D). As can be seen, the pattern of results in the habituation–IQ studies (Fig. 1) looks highly similar to results simulated under Strategy 4 and an ES greater than 0 (Fig. 4; Panels B4 and C4). The funnel plot from Alfieri et al. (2011; educational psychology) resembles that from Strategy 2 under a small ES (Panel B2). In Woodley’s meta-analysis (2011; evolutionary psychology), the overall effect is close to zero and Strategy 2 appears to be at play (Panel A2). In Farber and Doodlin’s meta-analysis (2011; psychotherapy works better with positive regard), samples sizes were small and there is an indication of the use of Strategy 3 (Panel B3). In Hallion and Ruscio’s meta-analysis (2011), the effects of cognitive bias modification on stress and anxiety appear small and based on too many underpowered studies (Panel A3 or B3). Correlations between relationship conflict and hostility of partners as studied by Woodin (2011) appear to be substantial but may also be inflated by publication bias and the use of small samples (Panel C3). In these research lines, additional studies with larger sample sizes are clearly welcome.

We also included in Figure 3 and Table 1 a recent meta-analysis on the predictive validity of the Implicit Association Test (IAT). The subset of studies that concerned racial discrimination is another example of an excess of significant results and funnel plot asymmetry. The results from the 32 studies collated by Greenwald, Poehlman, Uhlman, and Banaij (2009) are based on small samples sizes, considerable freedom in the analysis, and a high degree of faddism, all of which may conspire to bring about inflated effects (Ioannidis, 2005). Further studies with larger sample sizes should be added to the database to accurately determine IAT’s validity.

Without any clear rules concerning the use (and documentation) of multiple small studies and QRPs, strategic behaviors by researchers can lead literatures astray. The best way to separate the wheat from the chaff in psychology is to (a) end the pretense that small studies are definitive, (b) improve reporting standards, (c) start considering and publishing nonsignificant results, and (d) introduce a distinction between exploratory and confirmatory studies into journal policies (Wagenmakers, Wetzels, Borsboom, & van der Maas, 2011). The prevalence of underpowered studies in psychological research hints at the common use of Strategies 3 and 4 in which researchers conduct a series of small studies up to the point that one turns out to be significant. Sample sizes should be based on a priori power analyses that take into account the potential inflation of effects in earlier small studies. Later replications of published results and/or of statistical outcomes will be facilitated by including experimental material, raw data, and computer code (in the case of nonstandard analyses) as online supplements. In our view, researchers should be open about having not found what they looked for in early phases of research, while in the confirmatory phases, they should conduct studies that are possibly preregistered, sufficiently powerful, and analyzed in ways that are explicated in advance. The ideal paper then is not one with one or a few small studies with p values just below .05, but one in which all small pilot studies are reported in a meta-analytic summary and tested for homogeneity and/or moderation and in which one major study lends clear support. Small and underpowered studies may lead to biases of different kinds (Ledgerwood & Sherman, 2012) and some have even argued to simply exclude them from meta-analyses (Kraemer, Gardner, Brooks, & Yesavage, 1998).

We found indications of bias in nearly half of the psychological research lines we scrutinized. The ambition of players in the game of science does not always sit well with the goal of the scientific enterprise. Optimal strategies for individual researchers introduce biases that we can only counter by improving the rules of the game. The arbiters in the game (peer reviewers and editors) are in an ideal position to do so.