Empirical effect size benchmarks

empirical_es_benchmarks.json

[
  {"id":"BoscoCorrelationaleffectsize2015","abstract":"Effect size information is essential for the scientific enterprise and plays an increasingly central role in the scientific process. We extracted 147,328 correlations and developed a hierarchical taxonomy of variables reported in Journal of Applied Psychology and Personnel Psychology from 1980 to 2010 to produce empirical effect size benchmarks at the omnibus level, for 20 common research domains, and for an even finer grained level of generality. Results indicate that the usual interpretation and classification of effect sizes as small, medium, and large bear almost no resemblance to findings in the field, because distributions of effect sizes exhibit tertile partitions at values approximately one-half to one-third those intuited by Cohen (1988). Our results offer information that can be used for research planning and design purposes, such as producing better informed non-nil hypotheses and estimating statistical power and planning sample size accordingly. We also offer information useful for understanding the relative importance of the effect sizes found in a particular study in relationship to others and which research domains have advanced more or less, given that larger effect sizes indicate a better understanding of a phenomenon. Also, our study offers information about research domains for which the investigation of moderating effects may be more fruitful and provide information that is likely to facilitate the implementation of Bayesian analysis. Finally, our study offers information that practitioners can use to evaluate the relative effectiveness of various types of interventions.","author":[{"family":"Bosco","given":"Frank A."},{"family":"Aguinis","given":"Herman"},{"family":"Singh","given":"Kulraj"},{"family":"Field","given":"James G."},{"family":"Pierce","given":"Charles A."}],"container-title":"Journal of Applied Psychology","container-title-short":"JAP","DOI":"10.1037/a0038047","ISSN":"1939-1854 0021-9010","issue":"2","issued":{"date-parts":[[2015]]},"language":"en","page":"431-449","source":"APA PsycNET","title":"Correlational effect size benchmarks","type":"article-journal","volume":"100"},
  {"id":"ErpEstimatesbetweenstudyheterogeneity2017","abstract":"We present a data set containing 705 between-study heterogeneity estimates τ2 as reported in 61 articles published in Psychological Bulletin from 1990–2013. The data set also includes information about the number and type of effect sizes, the Q- and I2-statistics, and publication bias. The data set is stored in the Open Science Framework repository (https://osf.io/wyhve/) and can be used for several purposes: (1) to compare a specific heterogeneity estimate to the distribution of between-study heterogeneity estimates in psychology; (2) to construct an informed prior distribution for the between-study heterogeneity in psychology; (3) to obtain realistic population values for Monte Carlo simulations investigating the performance of meta-analytic methods. Funding statement: This research was supported by the ERC project “Bayes or Bust”.","accessed":{"date-parts":[[2018,4,7]]},"archive_location":"705 heterogeneity estimates from 61 articles published in Psychological Bulletin","author":[{"family":"Erp","given":"Sara","dropping-particle":"van"},{"family":"Verhagen","given":"Josine"},{"family":"Grasman","given":"Raoul P. P. P."},{"family":"Wagenmakers","given":"Eric-Jan"}],"container-title":"Journal of Open Psychology Data","container-title-short":"J. Open Psychol. Data","DOI":"10.5334/jopd.33","ISSN":"2050-9863","issue":"1","issued":{"date-parts":[[2017,8,17]]},"language":"en","source":"openpsychologydata.metajnl.com","title":"Estimates of between-study heterogeneity for 705 meta-analyses reported in <i>Psychological Bulletin</i> from 1990–2013","type":"article-journal","URL":"http://openpsychologydata.metajnl.com/articles/10.5334/jopd.33/","volume":"5"},
  {"id":"FunderEvaluatingeffectsize2019","abstract":"Effect sizes are underappreciated and often misinterpreted—the most common mistakes being to describe them in ways that are uninformative (e.g., using arbitrary standards) or misleading (e.g., squaring effect-size rs). We propose that effect sizes can be usefully evaluated by comparing them with well-understood benchmarks or by considering them in terms of concrete consequences. In that light, we conclude that when reliably estimated (a critical consideration), an effect-size r of .05 indicates an effect that is very small for the explanation of single events but potentially consequential in the not-very-long run, an effect-size r of .10 indicates an effect that is still small at the level of single events but potentially more ultimately consequential, an effect-size r of .20 indicates a medium effect that is of some explanatory and practical use even in the short run and therefore even more important, and an effect-size r of .30 indicates a large effect that is potentially powerful in both the short and the long run. A very large effect size (r = .40 or greater) in the context of psychological research is likely to be a gross overestimate that will rarely be found in a large sample or in a replication. Our goal is to help advance the treatment of effect sizes so that rather than being numbers that are ignored, reported without interpretation, or interpreted superficially or incorrectly, they become aspects of research reports that can better inform the application and theoretical development of psychological research.","accessed":{"date-parts":[[2019,6,17]]},"author":[{"family":"Funder","given":"David C."},{"family":"Ozer","given":"Daniel J."}],"container-title":"Advances in Methods and Practices in Psychological Science","container-title-short":"AMPPS","DOI":"10.1177/2515245919847202","ISSN":"2515-2459","issue":"2","issued":{"date-parts":[[2019,6,1]]},"language":"en","page":"156-168","source":"SAGE Journals","title":"Evaluating effect size in psychological research: sense and nonsense","title-short":"Evaluating effect size in psychological research","type":"article-journal","URL":"https://doi.org/10.1177/2515245919847202","volume":"2"},
  {"id":"GignacEffectsizeguidelines2016","accessed":{"date-parts":[[2017,1,6]]},"author":[{"family":"Gignac","given":"Gilles E."},{"family":"Szodorai","given":"Eva T."}],"container-title":"Personality and Individual Differences","container-title-short":"PID","DOI":"10.1016/j.paid.2016.06.069","ISSN":"0191-8869","issued":{"date-parts":[[2016,11]]},"language":"en","page":"74-78","source":"CrossRef","title":"Effect size guidelines for individual differences researchers","type":"article-journal","URL":"http://linkinghub.elsevier.com/retrieve/pii/S0191886916308194","volume":"102"},
  {"id":"Harrispolicyrelevantbenchmarksinterpreting2009","abstract":"The common reporting of effect sizes has been an important advance in education research in recent years. However, the benchmarks used to interpret the size of these effects—as small, medium, and large—do little to inform educational administration and policy making because they do not account for program costs. The author proposes an approach to establishing cost-effectiveness benchmarks rooted in an explicit economics-based decision-making framework and assumptions about the decision-making context. To be considered large, the ratio of effects to costs must be at least as large as the ratios for substitute interventions. Evidence related to class size, prekindergarten, and other interventions is discussed to illustrate the calculation of the cost-effectiveness ratios, how the evidence can be used to develop benchmarks, and how the benchmarks can be useful for researchers and policy makers. The development of benchmarks is intended to encourage cost-effectiveness analysis as a standard part of policy analysis, thereby providing more evidence to increase the validity of the benchmarks and, ultimately, improving policy decisions. Recent cost-effectiveness research in health care policy illustrates the potential value of cost-effectiveness benchmarks in education.","author":[{"family":"Harris","given":"Douglas N."}],"container-title":"Educational Evaluation and Policy Analysis","container-title-short":"EEPA","DOI":"10.3102/0162373708327524","ISSN":"0162-3737","issue":"1","issued":{"date-parts":[[2009,3,1]]},"language":"en","page":"3-29","source":"SAGE Journals","title":"Toward policy-relevant benchmarks for interpreting effect sizes: combining effects with costs","title-short":"Toward policy-relevant benchmarks for interpreting effect sizes","type":"article-journal","URL":"http://journals.sagepub.com/doi/abs/10.3102/0162373708327524","volume":"31"},
  {"id":"HemphillInterpretingmagnitudescorrelation2003","abstract":"Discusses empirical guidelines for interpreting the magnitude of correlation coefficients, a key index of effect size, in psychological studies. The author uses the work of J. Cohen (see record 1987-98267-000), in which operational definitions were offered for interpreting correlation coefficients, and examines two meta-analytic reviews (G. J. Meyer et al., see record 2001-00159-003; and M. W. Lipsey et al., see record 1994-18340-001) to arrive at the empirical guidelines.","author":[{"family":"Hemphill","given":"James F."}],"container-title":"American Psychologist","container-title-short":"AP","DOI":"10.1037/0003-066x.58.1.78","ISSN":"1935-990X(Electronic);0003-066X(Print)","issue":"1","issued":{"date-parts":[[2003]]},"language":"en","page":"78-79","source":"APA PsycNET","title":"Interpreting the magnitudes of correlation coefficients","type":"article-journal","volume":"58"},
  {"id":"HillEmpiricalbenchmarksinterpreting2008","abstract":"ABSTRACT—There is no universal guideline or rule of thumb for judging the practical importance or substantive significance of a standardized effect size estimate for an intervention. Instead, one must develop empirical benchmarks of comparison that reflect the nature of the intervention being evaluated, its target population, and the outcome measure or measures being used. This approach is applied to the assessment of effect size measures for educational interventions designed to improve student academic achievement. Three types of empirical benchmarks are illustrated: (a) normative expectations for growth over time in student achievement, (b) policy-relevant gaps in student achievement by demographic group or school performance, and (c) effect size results from past research for similar interventions and target populations. The findings can be used to help assess educational interventions, and the process of doing so can provide guidelines for how to develop and use such benchmarks in other fields.","author":[{"family":"Hill","given":"Carolyn J."},{"family":"Bloom","given":"Howard S."},{"family":"Black","given":"Alison Rebeck"},{"family":"Lipsey","given":"Mark W."}],"container-title":"Child Development Perspectives","container-title-short":"CHDP","DOI":"10.1111/j.1750-8606.2008.00061.x","ISSN":"1750-8606","issue":"3","issued":{"date-parts":[[2008,12,1]]},"language":"en","page":"172-177","source":"Wiley Online Library","title":"Empirical benchmarks for interpreting effect sizes in research","type":"article-journal","URL":"http://onlinelibrary.wiley.com/doi/10.1111/j.1750-8606.2008.00061.x/abstract","volume":"2"},
  {"id":"LockeGeneralizinglaboratoryfield1986","call-number":"HF5549 .G424 1986","collection-title":"The Issues in organization and management series","editor":[{"family":"Locke","given":"Edwin A."}],"ISBN":"978-0-669-09692-7","issued":{"date-parts":[[1986]]},"number-of-pages":"291","publisher":"Lexington Books","publisher-place":"Lexington, MA","source":"Library of Congress ISBN","title":"Generalizing from laboratory to field settings: research findings from industrial-organizational psychology, organizational behavior, and human resource management","title-short":"Generalizing from laboratory to field settings","type":"book"},
  {"id":"LovakovEmpiricallyderivedguidelines2017","abstract":"A number of recent research publications have shown that commonly used guidelines for interpreting effect sizes suggested by Cohen (1988) do not fit well with the empirical distribution of those effect sizes, and tend to overestimate them in many research areas. This study proposes empirically derived guidelines for interpreting effect sizes for research in social psychology, based on analysis of the true distributions of the two types of effect size measures widely used in social psychology (correlation coefficient and standardized mean differences). Analysis was carried out on the empirical distribution of 9884 correlation coefficients and 3580 Hedges’ g statistics extracted from studies included in 98 published meta-analyses. The analysis reveals that the 25th, 50th, and 75th percentiles corresponded to correlation coefficients values of 0.12, 0.25, and 0.42 and to Hedges’ g values of 0.15, 0.38, and 0.69, respectively. This suggests that Cohen’s guidelines tend to overestimate medium and large effect sizes. It is recommended that correlation coefficients of 0.10, 0.25, and 0.40 and Hedges’ g of 0.15, 0.40, and 0.70 should be interpreted as small, medium, and large effects for studies in social psychology. The analysis also shows that more than half of all studies lack sufficient sample size to detect a medium effect. This paper reports the sample sizes required to achieve appropriate statistical power for the identification of small, medium, and large effects. This can be used for performing appropriately powered future studies when information about exact effect size is not available.","accessed":{"date-parts":[[2017,12,5]]},"author":[{"family":"Lovakov","given":"Andrey"},{"family":"Agadullina","given":"Elena"}],"container-title":"PsyArXiv","container-title-short":"PsyArXiv","DOI":"10.17605/osf.io/2epc4","issued":{"date-parts":[[2017,11,27]]},"language":"en","note":"PsyArXiv: 2epc4","source":"psyarxiv.com","title":"Empirically derived guidelines for interpreting effect size in social psychology","type":"article-journal","URL":"https://psyarxiv.com/2epc4/"},
  {"id":"NyeHowbigare2018","abstract":"Recently, an effect size measure, known as dMACS, was developed for confirmatory factor analytic (CFA) studies of measurement equivalence. Although this index has several advantages over traditional methods of identifying nonequivalence, the scale and interpretation of this effect size are still unclear. As a result, the interpretation of the effect size is left to the subjective judgment of the researcher. To remedy this issue for other effect sizes, some have proposed guidelines for evaluating the magnitude of an effect based on the distribution of effect sizes in the literature. The goal of the current research was to develop similar guidelines for effect sizes of measurement nonequivalence and build on this work by also examining the practical importance of nonequivalence. Based on a review of past research, we conducted two simulation studies to generate distributions of effects sizes. Assuming the ideal scenario of invariant referent items, the results of these simulations were then used to develop empirical guidelines for interpreting nonequivalence and its effects on observed outcomes.","accessed":{"date-parts":[[2018,4,16]]},"author":[{"family":"Nye","given":"Christopher D."},{"family":"Bradburn","given":"Jacob"},{"family":"Olenick","given":"Jeffrey"},{"family":"Bialko","given":"Christopher"},{"family":"Drasgow","given":"Fritz"}],"container-title":"Organizational Research Methods","container-title-short":"ORM","DOI":"10.1177/1094428118761122","ISSN":"1094-4281","issued":{"date-parts":[[2018,3,15]]},"language":"en","page":"1094428118761122","source":"SAGE Journals","title":"How big are my effects? Examining the magnitude of effect sizes in studies of measurement equivalence","title-short":"How big are my effects?","type":"article-journal","URL":"https://doi.org/10.1177/1094428118761122"},
  {"id":"Patersonassessmentmagnitudeeffect2016","abstract":"This study compiles information from more than 250 meta-analyses conducted over the past 30 years to assess the magnitude of reported effect sizes in the organizational behavior (OB)/human resources (HR) literatures. Our analysis revealed an average uncorrected effect of r = .227 and an average corrected effect of ρ = .278 (SDρ = .140). Based on the distribution of effect sizes we report, Cohen’s effect size benchmarks are not appropriate for use in OB/HR research as they overestimate the actual breakpoints between small, medium, and large effects. We also assessed the average statistical power reported in meta-analytic conclusions and found substantial evidence that the majority of primary studies in the management literature are statistically underpowered. Finally, we investigated the impact of the file drawer problem in meta-analyses and our findings indicate that the file drawer problem is not a significant concern for meta-analysts. We conclude by discussing various implications of this study for OB/HR researchers.","accessed":{"date-parts":[[2016,2,5]]},"author":[{"family":"Paterson","given":"Ted A."},{"family":"Harms","given":"P. D."},{"family":"Steel","given":"Piers"},{"family":"Credé","given":"Marcus"}],"container-title":"Journal of Leadership & Organizational Studies","container-title-short":"JLOS","DOI":"10.1177/1548051815614321","ISSN":"1548-0518, 1939-7089","issue":"1","issued":{"date-parts":[[2016,2,1]]},"language":"en","page":"66-81","source":"jlo.sagepub.com","title":"An assessment of the magnitude of effect sizes: evidence from 30 years of meta-analysis in management","title-short":"An assessment of the magnitude of effect sizes","type":"article-journal","URL":"http://jlo.sagepub.com/content/23/1/66","volume":"23"},
  {"id":"PlonskyHowbigbig2014","abstract":"The calculation and use of effect sizes—such as d for mean differences and r for correlations—has increased dramatically in second language (L2) research in the last decade. Interpretations of these effects, however, have been rare and, when present, have largely defaulted to Cohen's levels of small (d = .2, r = .1), medium (.5, .3), and large (.8, .5), which were never intended as prescriptions but rather as a general guide. As Cohen himself and many others have argued, effect sizes are best understood when interpreted within a particular discipline or domain. This article seeks to promote more informed and field-specific interpretations of d and r by presenting a description of L2 effects from 346 primary studies and 91 meta-analyses (N > 604,000). Results reveal that Cohen's benchmarks generally underestimate the effects obtained in L2 research. Based on our analysis, we propose a field-specific scale for interpreting effect sizes, and we outline eight key considerations for gauging relative magnitude and practical significance in primary and secondary studies, such as theoretical maturity in the domain, the degree of experimental manipulation, and the presence of publication bias.","author":[{"family":"Plonsky","given":"Luke"},{"family":"Oswald","given":"Frederick L."}],"container-title":"Language Learning","container-title-short":"LL","DOI":"10.1111/lang.12079","ISSN":"1467-9922","issue":"4","issued":{"date-parts":[[2014,12,1]]},"language":"en","page":"878-912","source":"Wiley Online Library","title":"How big is “big”? Interpreting effect sizes in L2 research","title-short":"How big is “big”?","type":"article-journal","URL":"http://onlinelibrary.wiley.com/doi/10.1111/lang.12079/abstract","volume":"64"},
  {"id":"RichardOnehundredyears2003","abstract":"This article compiles results from a century of social psychological research, more than 25,000 studies of 8 million people. A large number of social psychological conclusions are listed alongside meta-analytic information about the magnitude and variability of the corresponding effects. References to 322 meta-analyses of social psychological phenomena are presented, as well as statistical effect-size summaries. Analyses reveal that social psychological effects typically yield a value of r equal to .21 and that, in the typical research literature, effects vary from study to study in ways that produce a standard deviation in r of .15. Uses, limitations, and implications of this large-scale compilation are noted. (PsycINFO Database Record (c) 2010 APA, all rights reserved) (journal abstract)","author":[{"family":"Richard","given":"F. D."},{"family":"Bond","given":"Charles F."},{"family":"Stokes-Zoota","given":"Juli J."}],"container-title":"Review of General Psychology","container-title-short":"RGP","DOI":"10.1037/1089-2680.7.4.331","ISSN":"1939-1552, 1089-2680","issue":"4","issued":{"date-parts":[[2003]]},"language":"en","page":"331-363","source":"CiteSeer","title":"One hundred years of social psychology quantitatively described","type":"article-journal","URL":"http://doi.apa.org/getdoi.cfm?doi=10.1037/1089-2680.7.4.331","volume":"7"},
  {"id":"Robertssystematicreviewpersonality2017","abstract":"The current meta-analysis investigated the extent to which personality traits changed as a result of intervention, with the primary focus on clinical interventions. We identified 207 studies that had tracked changes in measures of personality traits during interventions, including true experiments and prepost change designs. Interventions were associated with marked changes in personality trait measures over an average time of 24 weeks (e.g., d = .37). Additional analyses showed that the increases replicated across experimental and nonexperimental designs, for nonclinical interventions, and persisted in longitudinal follow-ups of samples beyond the course of intervention. Emotional stability was the primary trait domain showing changes as a result of therapy, followed by extraversion. The type of therapy employed was not strongly associated with the amount of change in personality traits. Patients presenting with anxiety disorders changed the most, and patients being treated for substance use changed the least. The relevance of the results for theory and social policy are discussed. (PsycINFO Database Record","author":[{"family":"Roberts","given":"Brent W."},{"family":"Luo","given":"Jing"},{"family":"Briley","given":"Daniel A."},{"family":"Chow","given":"Philip I."},{"family":"Su","given":"Rong"},{"family":"Hill","given":"Patrick L."}],"container-title":"Psychological Bulletin","container-title-short":"Psych. Bull.","DOI":"10.1037/bul0000088","ISSN":"1939-1455","issue":"2","issued":{"date-parts":[[2017,2]]},"language":"eng","page":"117-141","PMID":"28054797","source":"PubMed","title":"A systematic review of personality trait change through intervention","type":"article-journal","volume":"143"},
  {"id":"SteelImprovingmetaanalyticassessment2015","abstract":"Meta-analytic estimation of effect size variance is critical for determining the degree to which a relationship or finding generalizes across contexts. In most meta-analyses, population effect size variability is estimated by subtracting expected sampling error variance from observed variance, using only information from a limited set of available studies. We propose an improved Bayesian variance estimation technique that incorporates findings from previous meta-analytic research through an informed prior distribution of likely levels of effect size variance. The logic of exchangeability as a conceptual foundation for using an informed prior is explicated. On the basis of Monte Carlo simulations, we find the traditional method of meta-analytic variance estimation the most biased and least accurate technique across all sizes of meta-analyses considered. The Bayesian methodology incorporating an informed prior proved to be the most accurate and overall least biased of all estimation methods. Conceptual advantages and limitations that must be taken into account when incorporating an informed prior to estimate variability of effect sizes in a meta-analysis are also discussed.","accessed":{"date-parts":[[2016,10,21]]},"author":[{"family":"Steel","given":"Piers D. G."},{"family":"Kammeyer-Mueller","given":"John"},{"family":"Paterson","given":"Ted A."}],"container-title":"Journal of Management","container-title-short":"JOM","DOI":"10.1177/0149206314551964","ISSN":"0149-2063, 1557-1211","issue":"2","issued":{"date-parts":[[2015]]},"language":"en","page":"718-743","source":"jom.sagepub.com","title":"Improving the meta-analytic assessment of effect size variance with an informed Bayesian prior","type":"article-journal","URL":"http://jom.sagepub.com/content/early/2014/09/29/0149206314551964","volume":"41"},
  {"id":"Szucs2016EmpiricalAssessment","abstract":"We have empirically assessed the distribution of published effect sizes and estimated power by extracting more than 100,000 statistical records from about 10,000 cognitive neuroscience and psychology papers published during the past 5 years. The reported median effect size was d=0.93 (inter-quartile range: 0.64-1.46) for nominally statistically significant results and d=0.24 (0.11-0.42) for non-significant results. Median power to detect small, medium and large effects was 0.12, 0.44 and 0.73, reflecting no improvement through the past half-century. Power was lowest for cognitive neuroscience journals. 14% of papers reported some statistically significant results, although the respective F statistic and degrees of freedom proved that these were non-significant; p value errors positively correlated with journal impact factors. False report probability is likely to exceed 50% for the whole literature. In light of our findings the recently reported low replication success in psychology is realistic and worse performance may be expected for cognitive neuroscience.","author":[{"family":"Szucs","given":"Denes"},{"family":"Ioannidis","given":"John PA"}],"DOI":"10.1101/071530","issued":{"date-parts":[[2016,8,25]]},"language":"en","note":"bioRxiv: 071530","publisher":"bioRxiv","source":"bioRxiv","title":"Empirical assessment of published effect sizes and power in the recent cognitive neuroscience and psychology literature","type":"article","URL":"http://biorxiv.org/content/early/2016/08/25/071530"},
  {"id":"SzucsEmpiricalassessmentpublished2017","abstract":"Author summary Biomedical science, psychology, and many other fields may be suffering from a serious replication crisis. In order to gain insight into some factors behind this crisis, we have analyzed statistical information extracted from thousands of cognitive neuroscience and psychology research papers. We established that the statistical power to discover existing relationships has not improved during the past half century. A consequence of low statistical power is that research studies are likely to report many false positive findings. Using our large dataset, we estimated the probability that a statistically significant finding is false (called false report probability). With some reasonable assumptions about how often researchers come up with correct hypotheses, we conclude that more than 50% of published findings deemed to be statistically significant are likely to be false. We also observed that cognitive neuroscience studies had higher false report probability than psychology studies, due to smaller sample sizes in cognitive neuroscience. In addition, the higher the impact factors of the journals in which the studies were published, the lower was the statistical power. In light of our findings, the recently reported low replication success in psychology is realistic, and worse performance may be expected for cognitive neuroscience.","accessed":{"date-parts":[[2017,11,6]]},"author":[{"family":"Szucs","given":"Denes"},{"family":"Ioannidis","given":"John P. A."}],"container-title":"PLOS Biology","container-title-short":"PLOS Bio.","DOI":"10.1371/journal.pbio.2000797","ISSN":"1545-7885","issue":"3","issued":{"date-parts":[[2017,3,2]]},"language":"en","page":"e2000797","source":"PLoS Journals","title":"Empirical assessment of published effect sizes and power in the recent cognitive neuroscience and psychology literature","type":"article-journal","URL":"http://journals.plos.org/plosbiology/article?id=10.1371/journal.pbio.2000797","volume":"15"},
  {"id":"Taylor2018InvestigatingScience","abstract":"A priori power analyses allow researchers to estimate the number of participants needed to detect the effects of an intervention. However, power analyses are on...","accessed":{"date-parts":[[2020,4,28]]},"author":[{"family":"Taylor","given":"Joseph A."},{"family":"Kowalski","given":"Susan M."},{"family":"Polanin","given":"Joshua R."},{"family":"Askinas","given":"Karen"},{"family":"Stuhlsatz","given":"Molly A. M."},{"family":"Wilson","given":"Christopher D."},{"family":"Tipton","given":"Elizabeth"},{"family":"Wilson","given":"Sandra Jo"}],"container-title":"AERA Open","issue":"3","issued":{"date-parts":[[2018,8,9]]},"language":"en","title":"Investigating science education effect sizes: implications for power analyses and programmatic decisions","title-short":"Investigating science education effect sizes","type":"article-journal","URL":"https://journals.sagepub.com/doi/10.1177/2332858418791991","volume":"4"},
  {"id":"VanhoveReconcilingtwodisciplines2015","abstract":"A strong preference for field research exists in the organisational sciences. However, it is unclear whether or under what conditions this is warranted. To examine this issue we conducted a second?order meta?analysis of 203 lab?field pairs of meta?analytic effects representing a diverse range of work?related relationships. As expected, results showed a larger effect for lab (r?=?.25) than for field research (r?=?.14). However, the correspondence between the rank?order of effect sizes for relationships assessed in lab settings and matched effects assessed in field settings was weaker (r?=?.61) than previous estimates from related areas of research. Moderators of lab?field effect size magnitude and rank?order correspondence were tested. Effect size magnitudes from the lab and field were most similar when lab studies used correlational designs, when using psychological state and trait (as opposed to demographic or workplace characteristic) variables as predictors, and when assessing attitudinal outcomes. Lab?field rank?order correspondence was strongest when testing psychological state and workplace characteristic predictors and when assessing attitudinal and decisional outcomes. Findings offer recommendations for interpreting primary lab and field effects and inform evaluations of ?when? findings from lab and field studies are likely to align.","accessed":{"date-parts":[[2018,4,5]]},"author":[{"family":"Vanhove","given":"Adam J."},{"family":"Harms","given":"Peter D."}],"container-title":"Applied Psychology","container-title-short":"Appl. Psychol.","DOI":"10.1111/apps.12046","ISSN":"0269-994X","issue":"4","issued":{"date-parts":[[2015,9,16]]},"language":"en","page":"637-673","source":"onlinelibrary.wiley.com (Atypon)","title":"Reconciling the two disciplines of organisational science: a comparison of findings from lab and field research","title-short":"Reconciling the two disciplines of organisational science","type":"article-journal","URL":"https://onlinelibrary.wiley.com/doi/full/10.1111/apps.12046","volume":"64"},
  {"id":"WiernikEmpiricalbenchmarksinterpreting2017","abstract":"Generalization in meta-analyses is not a dichotomous decision (typically encountered in papers using the <i>Q</i> test for homogeneity, the 75% rule, or null hypothesis tests). Inattention to effect size variability in meta-analyses may stem from a lack of guidelines for interpreting credibility intervals. In this commentary, we describe two methods for making practical interpretations and determining whether a particular SDρ represents a meaningful level of variability.","accessed":{"date-parts":[[2017,9,1]]},"author":[{"family":"Wiernik","given":"Brenton M."},{"family":"Kostal","given":"Jack W."},{"family":"Wilmot","given":"Michael P."},{"family":"Dilchert","given":"Stephan"},{"family":"Ones","given":"Deniz S."}],"container-title":"Industrial and Organizational Psychology","container-title-short":"IOP","DOI":"10.1017/iop.2017.44","ISSN":"1754-9426, 1754-9434","issue":"3","issued":{"date-parts":[[2017,9]]},"language":"en","page":"472-479","source":"Cambridge Core","title":"Empirical benchmarks for interpreting effect size variability in meta-analysis","type":"article-journal","URL":"https://doi.org/10/ccnv","volume":"10"}
]