Prevalence of disclosed and undisclosed financial conflicts of interest among systematic review authors regarding the management of proximal humerus fractures
Cole Verble, B.S., Office of Medical Student Research, Oklahoma State University Center for Health Sciences, Tulsa, Oklahoma Matthew C. Ferrell, B.S., Office of Medical Student Research, Oklahoma State University Center for Health Sciences, Tulsa, Oklahoma Arjun K. Reddy, B.A., Office of Medical Student Research, Oklahoma State University Center for Health Sciences, Tulsa, Oklahoma J. Michael Anderson, B.S., Office of Medical Student Research, Oklahoma State University Center for Health Sciences, Tulsa, Oklahoma Michael Weaver M.S., Kansas City University of Medicine and Biosciences, College of Osteopathic Medicine, Joplin, Missouri Micah Hartwell PhD., Office of Medical Student Research, Oklahoma State University Center for Health Sciences, Tulsa, Oklahoma Matt Vassar, PhD, Office of Medical Student Research, Oklahoma State University Center for Health Sciences, Tulsa, Oklahoma
Funding: Development of this protocol and study was funded by the Oklahoma State University Center for Health Sciences Presidential Mentor-Mentee Research Fellowship Grant
Conflicts of Interest: Dr. Vassar reports receipt of funding from the National Institute on Drug Abuse, the National Institute on Alcohol Abuse and Alcoholism, the US Office of Research Integrity, Oklahoma Center for Advancement of Science and Technology, and internal grants from Oklahoma State University Center for Health Sciences — all outside of the present work. Dr. Hartwell has grant funding from the National Institutes of Justice, unrelated to the current work.
Abstract: Background: A systematic review is an important evidence synthesis technique used to collate results from individual studies, such as treatments for proximal humerus fractures. It is necessary to minimize bias in systematic reviews, including financial COIs, which have been shown to result in unreliable assessments of credibility.
Objective : The aim of this study was to characterize the influence of financial bias on the results and conclusions of systematic reviews of proximal humerus fracture treatments and to characterize the nature of disclosed and undisclosed COIs.
Methods: Ovid MEDLINE and Ovid Embase databases were searched to locate systematic reviews covering proximal humerus fracture treatments. Following these searches, title and abstract screening was performed in a duplicate, masked fashion. Data from the final reviews were extracted in a triplicate manner. The data from the final reviews included various author and article characteristics. These characteristics can be found under the Data extraction paragraph. All authors were screened for non-disclosed COIs.
Results: We found no relationship between authorial COI and the results and conclusions of the systematic reviews. Among the 17 included systematic reviews, 7 (41.2%) had at least one non-disclosed COI. Of the 7 reviews with a non-disclosed COI, 2 (28.6%) were found to have a high risk of bias.
Conclusions :Findings from this study have limited generalizability due to our small sample size. More studies are needed to fully elucidate the effect of financial bias on the results and conclusions of systematic reviews.
Keywords: proximal humerus fractures, conflicts of interest, financial conflicts of interest, systematic reviews, proximal humerus
Introduction : Proximal humerus fractures account for 6% of all fractures and are commonly seen in patients with osteoporotic disease following low impact falls.1–3 Proximal humerus fractures are commonly managed conservatively, as most injuries heal without more invasive intervention. However, in more severe cases, the best course of treatment is subject to debate, and has been the topic of discussion throughout the orthopedic literature.4–6 With an aging population, the incidence of osteoporotic fractures – including proximal humerus fractures – are expected to increase in the coming years.7,8 Given the expected rise in disease burden, it is essential orthopedic surgeons critically appraise research outcomes, as well as the overall quality of evidence, from studies regarding the treatment of proximal humerus fractures.
The American Academy of Orthopedic Surgeons (AAOS) consider systematic reviews of Level 1 randomized controlled trials (RCTs) among the highest level of evidence in clinical research.9,10 Given their spot atop the hierarchy of evidence, systematic reviews often serve as the foundation upon which clinical practice guideline recommendations are based. Despite the potential utility of systematic reviews in helping achieve optimal patient outcomes, previous studies have demonstrated that systematic reviews published in the orthopedic literature are not free of potential forms of bias. For example, Scott, et al., found a large percentage of studies published in high impact orthopedic journals failed to assess for publication bias; and, when studied, nearly one-third demonstrated evidence of publication bias.11 Systematic reviews failing to account for sources of bias may result in misguided clinical decisions, with the potential to affect patient care.12 Another potential source of bias that may call into question the validity of study outcomes is the presence of conflicts of interest (COI), among systematic review authors.
According to the International Committee of Medical Journal Editors (ICMJE), a COI exists when “... a professional judgment concerning a primary interest (such as patients' welfare or the validity of research) may be influenced by a secondary interest (such as financial gain)”.13 These industry relationships carry the potential to influence the nature of study outcomes, and calls to question the reliability and validity of such results. Take for example one study which found authors disclosing significant COIs with pharmaceutical industries were more likely to report favorable outcomes compared to authors without a COIs. Given the potential harm these COIs may have in medicine, further investigation into the extent that systematic review authors disclose COIs – as well as determine the influence these COIs have on the nature of outcomes reported in systematic reviews in the orthopedic literature – is warranted.
Thus, the aim of this study was to characterize the nature of disclosed and undisclosed COIs of systematic review authors, specifically with regard to the treatment of proximal humerus fracture. Additionally, we sought to determine whether the direction of narrative results and/or conclusions from these reviews are influenced by authors receiving significant financial compensation.
Methods: Transparency, Reproducibility, and Reporting Institutional review board oversight was not required for this cross-sectional study as it did not involve human subjects.14 To facilitate reproducibility and transparency of our results, we have supplied the study protocol, materials, and data sets on Open Science Framework.15 The Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines were used in the process of drafting this manuscript.16 Additionally, we referred to Murad and Wang's guidelines for conducting meta-epidemiological research.17 Deviations from study protocol Due to the essence of our final data, we could not determine whether industry-funded systematic reviews were more or less likely to find positive results and conclusions with regard to an intervention than systematic reviews funded by other sources (including those that did not receive funding support). Not a single industry-funded systematic review was found in our sample, which precluded us from determining whether associations in outcome reporting may exist. This topic may need further investigation.
Study Objectives Our primary objectives were to (1) determine the frequency of COIs (both disclosed undisclosed) among authors of systematic reviews and meta-analyses focusing on the treatment of proximal humerus fractures; and (2) determine if author COI affects the overall narrative results and conclusions. As secondary objectives, we evaluated (1) whether an association exists between risk of bias and COI among review authors and (2) whether the presence of study sponsorship influenced reported results and conclusions.
Search Strategy MEDLINE (Ovid) and Embase (Ovid) databases were searched using the same strategies developed by a systematic review librarian, as that outlined in a previous protocol, for a prior study within our research group.18 The search was performed June 2, 2020 to identify systematic reviews with or without meta-analyses using a search strategy in Supplemental File 1. Following the execution of these searches, the resulting records were uploaded to Rayyan (https://rayyan.qcri.org/), a systematic review platform for title and abstract screening.
Screening The initial results for the search strategy were screened from a previous study to include systematic reviews concerning treatments for proximal humerus fractures. Specific screening criteria can be found in the protocol.15 We further refined our inclusion criteria for screening by AR, CV, and MF in a masked, triplicate manner based upon these additional criteria which are outlined below. All discrepancies were resolved in a group meeting after the screening process was completed.
Eligibility Criteria A study was deemed eligible for inclusion if it (1) met the PRISMA-P definition of a systematic review and/or meta-analysis; (2) was a head-to-head comparison of one treatment to either another treatment (or combination) or a placebo/standard of care; (3) focused on treatments for proximal humerus fractures; (4) was published between September 1, 2016 and June 2, 2020; (5) was published in the English language; and (5) synthesized data from systematic reviews using human data. As prespecified in the study protocol, if more than 200 studies were eligible for inclusion, studies were uploaded to STATA for randomization. Data were subsequently extracted from the first 200 randomized studies.
Training All investigators completed training, online and in-person, before study commencement. Training was recorded and is available online.15 In short, this training session consisted of the study design and objectives, study materials, and a step-by-step explanation of how to perform data extraction using an example systematic review.
Data extraction Three authors (AR, CV, and MF) were assigned with data extraction. Data were pulled independently in a masked, triplicate fashion using a pilot-tested Google form. Full-text of each systematic review or meta-analysis was examined and the following data items were extracted: (1) PubMed identification number and/or DOI, (2) journal name, (3) publication date, (4) name of authors, compared treatment interventions, (5) first and last author affiliation(s), (6) source of funding, (7) full COI statement, (8) risk of bias assessment within the systematic review or meta-analysis, (9) the verbatim risk of bias statement, (10) whether systematic review author(s) were also an author on any of the primary studies included in the review, (11) amount of self-cited primary studies, (12) the systematic reviews primary outcome or the first outcome reviewed, (13) whether an overall pooled effect estimate was calculated, (14) pooled effect estimate for the primary outcome, (15) type of calculated pooled effect estimate (eg, mean difference, risk ratio, odds ratio), (16) statistical significance of pooled effect estimate, (17) the primary outcomes favorability of pooled effect estimate in regards to the primary outcome, and (18) whether narrative results and conclusions favored the comparison or treatment group (e.g., placebo, standard of care, control). For the purpose of this study, “conclusion” was used to represent a review’s discussion and conclusion.
Favorability of narrative results and conclusions We reviewed the favorability of the narrative results and conclusions by designating them as favorable, unfavorable, or mixed/inconclusive. While reviewing results we considered favorable when only positive results reported for all study populations. Unfavorable was assigned when only negative results reported for at least one study population. Mixed/inconclusive was assigned if both positive and negative results reported for the study populations within the narrative. While reviewing conclusions, favorable was assessed when authors reported either explicitly or implicitly in favor of the target intervention. Conversely unfavorable was assessed if the authors explicitly or implicitly favored the control group. Mixed/inconclusive was assessed if we are unable to meet criteria for favorable or unfavorable (e.g., reporting negative population outcome but positive subgroup analysis).
Identification of undisclosed conflicts of interest Our stepwise strategy of the search for undisclosed COI is located in Figure 1. For this process, we modified the methodology by Mandrioli et al,19 by incorporating 3 additional databases – the Open Payments database, Dollars for Profs, and the United States Patent and Trademark Office (USPTO). Table 1 describes each database. All authors for each systematic review were searched for undisclosed COI, regardless of disclosed COIs. Database-specific search strings were generated by a custom program created by MW using the Python programming language (Python Software Foundation, https://www.python.org/) for Google Patents, the United States Patent and Trademark Office (USPTO) Database, and PubMed to ensure reproducibility and accuracy of data extraction. We chose to limit searches of patents to 10 years prior to the review's publication due to the longevity of patents. If we were unable to verify if the patents from our searches belonged to the author for whom we searched, we did not consider it an undisclosed COI. PubMed searches for each author reviewed the conflict statement of all of the authors published works up to 36 months prior to the publication of the original review. If more than 10 manuscripts were found during the initial PubMed search, then random numbers were assigned to all PubMed manuscripts returned for an author. After, data was extracted from the first 10 randomized manuscripts starting with the lowest number. AR, CV, and MF all generated their own random samples to broaden the search strategy. The search process was continued until it reached its conclusion or an undisclosed conflict of interest was found. This termination process was also used by Mandrioli et al.19