What is new?
Key findings- •
An extensive list of categories and examples of selective inclusion and reporting in randomized controlled trials (RCTs) and systematic reviews of RCTs was collated. Few empirical studies investigating the extent of bias associated with selective inclusion or reporting in systematic reviews of RCTs exist.
What this adds to what was known?- •
To our knowledge, this is the first systematic review of reports describing empirically investigated or hypothetical examples of selective inclusion and reporting in RCTs and systematic reviews of RCTs.
What is the implication and what should change now?- •
Trialists and systematic reviewers need to be aware of the scenarios in which they may inadvertently introduce bias through selective inclusion or reporting of results.
- •
Increasing trialists' and systematic reviewers' awareness of these examples may minimize their occurrence.
- •
More methodological research is needed to investigate the magnitude of bias resulting from different examples of selective inclusion and reporting, particularly at the systematic review level.
Systematic reviews of randomized controlled trials (RCTs) of health care interventions are used by clinicians to inform their treatment options, clinical practice guideline developers to formulate recommendations, and funding bodies to determine whether there is a justification for further research [1], [2], [3]. The success of these activities may be compromised when the methods used throughout the review process result in bias, defined as any systematic error that can over- or underestimate an intervention effect [4]. To inform systematic reviewers about methods that minimize bias in the context of systematic reviews, methodologists have developed lists of problematic practices, for example, searching only a single electronic bibliographic database or screening studies for eligibility by only a single reviewer [1], [4], [5], [6], [7], [8]. One such practice that has gained attention in recent years is selective reporting, defined as the selection of a subset of outcomes and analyses to report in a publication [9], [10], [11], [12].
Selective reporting can occur in various ways in both RCTs and systematic reviews of RCTs. In RCTs, examples include the nonreporting of outcomes that have been measured and analyzed or the partial reporting of results (e.g., reporting an effect estimate with no measure of variation when the result is nonsignificant) [10], [12], [13], [14], [15], [16], [17]. When the way in which outcomes and analyses are reported is based on the results (e.g., statistical significance, magnitude, or direction of effect), this is known as selective reporting bias [9], [10], [11]. In systematic reviews, when multiplicity of outcome data is available in RCTs, systematic reviewers may choose to include only a subset of this data. For example, if data for the outcome depression are reported in a journal article based on two measurement scales, each at three time points, the systematic reviewers may choose to only include the data from one scale at one time point. This practice is not always problematic, such as when the choice of outcome data is prespecified [18], [19]. However, when the choice of outcome data to include is based on the results (which we refer to as “selective inclusion”), this can introduce bias. After inclusion of outcome data, outcomes and analyses may be selectively reported in systematic reviews in the same way as occurs in RCTs (e.g., selecting which outcomes and meta-analytic effect estimates to report based on the results) [20], [21]. Both selective inclusion and reporting may over- or underestimate meta-analytic results [9], [10], [11], [19], limit interpretation, and mislead users about the importance of particular outcomes [13], [21]. Fig. 1 illustrates the levels at which selective reporting in RCTs, selective inclusion in systematic reviews, and selective reporting in systematic reviews can occur. The example depicts a scenario in which multiple measurement instruments of depression are used with different transformations of the outcomes (final and change from baseline values).
There are many additional ways in which outcomes and analyses can be selectively included or reported [1], [7], [12], [16], [22], [23]. To date, there has been no review of the literature describing these practices. Collating such a list has multiple purposes: it increases the trialists' and systematic reviewers' awareness of possible types of selective reporting which may occur at the RCT level, it highlights how systematic reviewers may inadvertently introduce bias through the selective inclusion of results or misinform users of systematic reviews through selective reporting of results, it helps to identify where empirical research may be required to investigate the prevalence and impact of potential sources of bias, and it guides methodological advice regarding how to minimize these practices. The aims of this research were therefore to (1) collate and categorize the ways in which selective inclusion and reporting can occur in RCTs and systematic reviews and (2) identify the types of selective inclusion or reporting that have been researched in empirical studies investigating such bias. To meet these aims, we conducted a systematic review that included reports describing examples of selective inclusion or reporting. We then categorized the identified examples and made judgments about whether examples reported at one level (e.g., selective reporting in RCTs) could hypothetically apply to other levels (e.g., at either the selective inclusion or reporting in systematic reviews levels or both). It was beyond the scope of this review to synthesize the results of empirical studies investigating selective inclusion or reporting—a systematic review of empirical studies investigating selective reporting in RCTs exists [23], and we are currently synthesizing the results of empirical studies investigating selective inclusion and reporting in systematic reviews [24]. This work will be reported elsewhere.
