Frequency of Use of Clinical Trials Registries in Systematic Reviews and Meta-Analyses in Primary Care Journals


Julie A. Dionne DO, Lora Cotton DO, Christopher Thurman DO, Matt Vassar PhD



Abstract

Objective: To evaluate the use of clinical trials registries in published primary care systematic reviews and meta-analyses.

Background: Systematic reviews and meta-analyses in primary care medicine that do not report searching for unpublished data may be prone to publication bias. This source of bias occurs when systematic review evidence is based solely on a published body of literature and does not account for studies where results failed to achieve statistical significance and were therefore less likely to be published. One method to address publication bias is to perform a comprehensive search that includes the use of clinical trials registry searches in systematic reviews and meta-analyses published in the top 10 primary care journals over the past 8 years.

Methods: Systematic reviews and meta-analyses were retrieved from the top 10 primary care medicine journals published between January 1st, 2008 and December 31st, 2015 using a highly sensitive search string designed in collaboration with a medical librarian on staff at the Oklahoma State University Center for Health Sciences. We performed a meta-epidemiological study to evaluate the frequency of clinical trials registry searches reported in these systematic reviews and meta-analyses over time.

Results: A total of 205 articles retrieved from PubMed were included for analysis. We found that of the 205 systematic reviews and meta-analyses, only 8.3% (17/205) searched clinical trials registries for unpublished data.

Conclusions: A large portion of systematic reviews and meta-analyses in primary care literature do not search clinical trials registries for unpublished data. Omission of unpublished data from analysis may lead to publication bias and reduced validity of the true effect sizes of interventions.

Introduction

Systematic reviews and meta-analyses are utilized to incorporate data from many similar studies to help physicians and medical organizations make informed, evidenced-based decisions regarding patient care and general health policy. These specific types of studies have become increasingly important in healthcare settings. Clinicians do not have time to search for all articles on a specific topic, read them, draw conclusions, and put into practice what would be found within that search. Systematic reviews and meta-analyses are good tools for clinicians to keep up-to-date within their field and are often used as a starting point for developing clinical practice guidelines.1 For example, a recent systematic review was completed for the U.S. Preventive Services Task Force (USPSTF) to develop new recommendations on the use of statin therapy for prevention of cardiovascular disease (CVD) in adults who have no prior cardiovascular events. The following was the conclusion of this systematic review: in adults at increased CVD risk but without prior CVD events, statin therapy is associated with reduced risk of all-cause and cardiovascular mortality and CVD events; benefits appear to be present across diverse demographic and clinical subgroups; greater absolute benefits are found in patients at higher baseline risk and do not appear to be restricted to patients with marked hyperlipidemia.2 From systematic reviews such as this one, recommendations are made to alter current therapy guidelines or institute new guidelines. However, there is a hidden limitation to these types of studies. If during the search phase of the systematic review only articles that have been published are searched, then there is an opportunity for bias to be present. This is known as publication bias. This particular systematic review performed for the USPSTF only searched published articles and did not utilize a search of any clinical trials registries for ongoing or completed but unpublished studies. Thus, there may be an overrepresentation of statistically significant outcomes and an underrepresentation of conflicting or non-significant results.

If authors of systematic reviews and meta-analyses continue to search only published studies, there will be a limitation in the accuracy of their conclusions because authors are less likely to submit research with statistically non-significant results. Furthermore, journal editors are less likely to publish such findings. If treatment decisions are often based on published literature, then that literature should include all available data.3 A solution to this problem is searching clinical trials registries for unpublished studies during the search phase of all systematic reviews and meta-analyses.

In the past, registering studies with a clinical trials registry was voluntary and not widely used. Thus, many studies were performed with negative or less than significant results, and it is unlikely the larger scientific community has access to this data. To address this issue, in 2005 the International Committee of Medical Journal Editors mandated that journals in its membership network require researchers to register all trials with a public trials registry in order to be eligible for publication in their journals.4 In 2007, the U.S. government passed the U.S. Food and Drug Administration Amendments Act, which specifically directed researchers conducting human clinical trials to register their trials with www.clinicaltrials.gov, which is the U.S. trials registry.5 Furthermore, in 2015 the World Health Organization (WHO) released a statement supporting the public disclosure of clinic trial results and outlined timeframes for reporting completed studies and steps to improve linkages between clinical trials registry entries and their published results.6

Even with these positive steps towards increasing the utilization of trials registries, there is evidence that these registries are underused in systematic reviews and meta-analyses. This study focuses specifically on the frequency that clinical trials registry searches are used in primary care systematic reviews and meta-analyses.

Materials and Methods

Article Selection

A PubMed search for systematic reviews and meta-analyses was conducted using the following search string: ("Meta-Analysis"[Publication Type] OR (systematic[ti] AND review[ti]) OR metaanal*[ti] OR meta-anal*[ti]) AND ("1544 1709"[Journal] OR 0960-1643[Journal] OR 0002-838X[Journal] OR 1557-2625[Journal] OR 1471-2296[Journal] OR 0263-2136[Journal] OR 0281-3432[Journal] OR 0008-350X[Journal] OR 0742-3225[Journal] OR 0091-3847[Journal]) AND ("2008/01/01"[PDat] : "2015/12/31"[PDat]).  The search strategy was adapted from a previously published search strategy shown to be sensitive for the retrieval of systematic reviews and meta-analyses.7 Journals were selected based on the top ten primary health care journals in ISI Web of Knowledge Journal Citation Reports ranked by the 5-Year Impact Factor. These journals are the American Family Physician, Annals of Family Medicine, BMC Family Practice, British Journal of General Practice, Canadian Family Physician, Family Medicine, Family Practice, Journal of American Board of Family Medicine, The Physician and Sportsmedicine, and the Scandinavian Journal of Primary Healthcare.  PubMed was selected because it catalogues all ten journals included in the study. Journals were selected based on the highest 2014 h5-index of Google Scholar, primary care subcategory. The search was conducted on March 7, 2016.

Our PubMed search produced 222 results. All full-text articles were retrieved and screened using PubMed and EndNote. After removal of duplicate articles and assessment of all full-text articles, the number of studies in our review was 205 (see Figure 1).


Review process and coding

Definitions of systematic reviews and meta-analyses were provided to the primary coder and the co-authors. For the purposes of this study, a systematic review was defined as the following: has a clearly stated topic of review, an explicit search strategy, inclusion and exclusion criteria, and at least one rigorous primary study.8 A Meta-analysis requires a well-executed systematic review, a comprehensive search strategy of one or more electronic databases, a quality assessment of relevant studies with explicit and objective criteria for inclusion or rejection, calculating effect sizes – odds ratio vs. risk ratio (relative risk), checking for publication bias – funnel plot (displays the studies included in the meta-analysis in a plot of effect size against a sample size), sensitivity analyses, and a presentation of the findings using a Forest Plot.9

The following list of clinical trials registries was used including the WHO’s Primary Registries listed in the WHO Registry Network: the Australian New Zealand Clinical Trials Registry (ANZCTR), Brazilian Clinical Trials Registry (ReBec), Chinese Clinical Trial Registry (ChiCTR), Clinical Research Information Service (CRIiS), Republic of Korea, Clinical Trials Registry – India (CTRI), Cuban Public Registry of Clinical Trials (RPCEC), EU Clinical Trials Register (EU-CTR), German Clinical Trials Register (DRKS), Iranian Registry of Clinical Trials (IRCT), Japan Primary Registries Network (JPRN), Thai Clinical Trials Registry (TCTR)s, The Netherlands National Trial Register (NTR), Pan African Clinical Trial Registry (PACTR), and Sri Lanka Clinical Trials Registry (SLCTR). For a clinical trials registry to qualify for the WHO’s list, it must meet specific criteria for content, quality, validity, accessibility, unique identification, technical capacity, and administration. In addition to the WHO’s primary list, the following clinical trials registries were also deemed acceptable: www.clinicaltrials.gov, www.centerwatch.com, www.controlled-trials.com. Systematic reviews and meta-analyses that attempted to search “other clinical trial registries” or private clinical trials registries were also deemed acceptable for this study. Of note, the Cochrane Central Register of Controlled Trials (CENTRAL) is a registry of published articles and as such does not qualify as a clinical trial registry. It was not included in trial registry coding in this study.

To ensure accurate coding, intra-rater agreement was assessed to determine consistency of coding by randomly selecting two subsets of 10 primary care journal articles and having co-authors (L.C. and C.T.) review them. Accuracy was found to be 100%. The following elements were coded: (a) title, (b) authors’ name(s), (c) year of publication, (d) journal name, (e) if the article met criteria for systematic review and/or meta-analysis, and (f) if a clinical trial registry was included/which trial registries were searched.

Results

We conducted a review of primary care journals to assess if clinical trials registries were included in the search of published systematic reviews or meta-analyses. This study did not meet the regulatory definition of human subject research as defined in 45 CFR 46.102(d) and (f) of the Department of Health and Human Services' Code of Federal Regulations and, therefore, was not subject to institutional review board oversight. We applied relevant Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines for systematic reviews and Statistical Analyses and Methods in the Published Literature (SAMPL) guidelines for reporting descriptive statistics.

This search of PubMed yielded 222 articles, which was narrowed to 205 articles after exclusion (Fig 1). As displayed in Figure 2, we examined the use of searches of clinical trials registries during the search process of the systematic reviews and meta-analyses.




Of the 205 articles, seventeen (8.3%) conducted a search in at least one clinical trials registry.




The registries that were searched were www.clinicaltrials.gov (n=7), www.controlled-trials.com (n=4), the WHO International Clinical Trials Registry Platform (ICTRP) (n=4), Australian New Zealand Clinical Trials Registry (ANZCTR) www.anzctr.org.au (n=1), and the National Research Register UK (n=2). Two private trial registries were searched: https://tamifluclinic.net and http://www.Rochetrials.com/trialresults.action (n=1). Seven of the seventeen systematic reviews and meta-analyses searched trials registries but did not list specific registries that were searched.

During the first two years of this study (2009 and 2010), only three articles reported searching clinical trials registries, while during the last two years (2014 and 2015), eight articles reported searching clinical trials registries. This equals a 160% increase over the course of the eight years studied.

Discussion

When performing a systematic review or meta-analysis, we propose that searching of ongoing clinical trials registries be regularly included in the search strategy to account for negative results and adverse effects that may not otherwise be available in published literature. Golder et al performed a systematic review comparing the reporting of adverse events in both published and unpublished studies of health care interventions.10 This systematic review found strong evidence that much of the information on adverse events remains unpublished. Studies performed comparing results with and without inclusion of unpublished data have shown varied results. In particular, Whittington et al analyzed published data from two trials of fluoxetine. The published data suggested a favorable risk-benefit profile for the treatment of depression in children and young adults. Unpublished data also supported this claim. Published data from a trial of paroxetine and two trials of sertraline suggested weakly positive risk-benefit profiles, but with the addition of unpublished data, paroxetine and sertraline had risks that outweighed the benefits. Whittington et al also analyzed unpublished data from trials of citalopram and venlafaxine which suggested unfavorable risk-benefit profiles.11 A study by Hart et al in 2012 added unpublished outcome data from trials obtained from the US Food and Drug Administration to published meta-analyses. This study documented that the addition of the unpublished data could change the magnitude of the effect size or even the statistical significance of the meta-analyses.12

If clinical trials registries contain nonsignificant findings, systematic reviews and meta-analyses can in turn incorporate those findings, and results will show a more comprehensive representation of the subject matter. This will lead to more reliable systematic reviews, meta-analyses, and treatment guidelines.

Remarkably, the results of this study show that most primary care systematic reviews and meta-analyses do not search clinical trials registries. Only seventeen of the 205 (8.3%) primary care articles reported searching any clinical trials registries. When compared to other specialties, primary care systematic reviews and meta-analyses searched clinical trials registries at a lower rate. A recent analysis by Bibens et al found that 18.5% of obstetric and gynecologic systematic reviews conducted a search of a clinical trials registry.13 Similar results were found in emergency medicine (19%).14 Sinnett et al searched neurologic literature and found only six percent included searches of clinical trials registries.15 Thirty-five percent of studies from high-impact general medical journals report searching clinical trials registries, more than four times the percentage found in primary care.16

Unfortunately, barriers exist to searching clinical trials registries. One such barrier was reported by Whittington et al, which was an unwillingness of pharmaceutical companies to share information regarding the unpublished data of antidepressants.11 Baudard et al found that results were frequently unavailable even if trials registries were searched.17 Although trials are required to register, reporting of results is often lagging or even absent.17

When unpublished data is not included in systematic reviews or meta-analyses, conclusions and recommendations may be less precise.10 This could adversely affect patient outcomes by impairing the physician’s ability to make treatment decisions based on inaccurate conclusions.

Conclusion

A large portion of systematic reviews and meta-analyses in primary care literature do not search clinical trials registries for unpublished data. Omission of unpublished data from analysis may lead to publication bias and reduced validity of the true effect sizes of interventions. In the small sampling seen in this study, it does appear that searching clinical trials registries is increasing but is not yet a routine practice. Barriers exist that deter authors from performing clinical trials registry searches, and this could be the focus of future research. The unpublished data found in the clinical trials registries must be included in all systematic reviews and meta-analyses in order to obtain accurate answers to clinical questions.

References

1. Gopalakrishnan S, Ganeshkumar P. Systematic reviews and met-analysis: understanding the best evidence in primary healthcare. Journal of Family Medicine and Primary Care 2013;Jan-Mar 2(1):9-14.

2. Chou R, Dana T, Blazina I, Daeges M, Bougatsos C, Grusing S, Jeanne TL. Statin use for the prevention of cardiovascular disease in adults: a systematic review for the U.S. Preventive Services Task Force. Evidence Synthesis No. 139. AHRQ Publication No. 14-05206-EF-2. Rockville, MD: Agency for Healthcare Research and Quality 2016.

3. Dickersin K. The existence of publication bias and risk factors for its occurrence. Journal of the American Medical Association 1990;23:1385-9.

4. De Angelis C, Drazen JM, Frizelle FA, Haug C, Hoey J, Horton R, et al. Clinical trial registration: a statement from the International Committee of Medical Journal Editors. New England Journal of Medicine 2004;351;1250-1.

5. Food and Drug Administration Amendments Act of 2007. US Public Law 110-85; 21 USC 301, September 27, 2007.

6. World Health Organization. WHO statement on public disclosure of clinical trial results. Available at http://www.who.int/ictrp/results/reporting/en/ . Retrieved July 12, 2017.

7. Montori VM, Wilczynski NL, Morgan D, Haynes RB. Optimal search strategies for retrieving systematic reviews from Medline: Analytical Survey. British Medical Journal 2005 Jan 6;330(7482):68.

8. Montori VM, Wilczynski NL, Morgan D, Haynes RB, et al. Systematic Reviews: A Cross-sectional Study of Locations and Citation Counts. BioMed Central Medicine 2003;1:2.

9. Crombie IK, Davies HT. What is meta-analysis? evidence-based medicine. April,2009;1-8. Available at https://fhs.mcmaster.ca/anesthesiaresearch/documents/Crombie2009Whatismeta-analysis_.pdf . Retrieved March 1, 2016.e2.

10. Golder S, Loke YK, Wright K, Normal G. Reporting of adverse events in published and unpublished studies of health care interventions: a systematic review. PLOS 2016.

11. Whittington CJ, Kendall T, Fonagy P, Cottrell D, Cotgrove A, Boddington E. Selective serotonin reuptake inhibitors in childhood depression: systematic review of published versus unpublished data. Lancet 2004;262:1341-45.

12. Hart B, Lundh A, Bero L. Effect of reporting bias on meta-analyses of drug trials: reanalysis of meta-analyses. BMJ2012;356:d7202.

13. Bibens ME, Chong AB, Vassar, M. Utilization of clinical trials registries in obstetrics and gynecology systematic reviews. Obstetrics and Gynecology 2016;127 (2):248-53.

14. Keil LG, Platts-Mills TF, Jones CW. Systematic reviews published in emergency medicine journals do not routinely search clinical trials registries: a cross-sectional analysis. Annals of Emergency Medicine 2015:66:424-27.e2.

15. Sinnett PM, Carr B, Cook G, Mucklerath H, Varney L, Weiher M, et al. Systematic reviewers in clinical neurology do not routinely search clinical trials registries. Public Library of Science One 2015;10:e0134596.

16. Jones CW, Keil LG, Weaver MA, Platts- Mills TF. Clinical trials registries are under-utilized in the conduct of systematic reviews: a cross-sectional analysis. Systematic Reviews 2014;3:126.

17. Gaudard M, Yavchitz A, Ravaud P, Perrodeau E, Boutron I. Impact of searching clinical trial registries in systematic reviews of pharmaceutical treatments: methodological systematic review and reanalysis of meta-analyses. BMJ2017;356:j448.