Carolyn M. Marshall, Oklahoma State University Center for Health Systems Innovation
Melissa Lay, DHA., Oklahoma State University Center for Health Systems Innovation
Corresponding Author:
Marjorie A. Erdmann
OSU Center for Health Sciences, Rm. 332, North Building.,
Tulsa, OK 74107,918-640-2006
Marjorie.erdmann@okstate.edu
Funding Statement: Oklahoma State University Center for Health Systems Innovation funded the project manager who ran the survey response effort at the hospital and gathered the metrics reported in this manuscript.
Conflicts of Interest/Disclosures: Carolyn Marshall is the Regional Representative for the Leapfrog Group. It is an advocacy role and unpaid. In this role she is asked to encourage non-participating hospitals to participate in the voluntary survey which raised the questions – what precisely the Leapfrog Group is asking hospitals to do and what effort does a first response take?
Abstract
Background: In pursuit of increasing consumer utilization of hospital safety information, The Leapfrog Group (LFG) asks hospitals to voluntarily self-report information through a survey after which LFG assesses responses’ achievement to LFG standards and publicly posts the hospital’s achievement levels. A substantial number of hospitals do not participate, including the vast majority of Oklahoma hospitals. A potential barrier is that administrators lack information to estimate “what it takes” to participate. Therefore, the authors measured a first-time effort and experience of LFG survey participation.
Method: Researchers at Oklahoma State University Center for Health Systems Innovation conducted a field study to discover the resources needed in personnel, project management, time, and effort to participate in the survey. The authors directly supported the first-time participation of a 199-bed hospital by providing them with a 0.25 FTE project manager and recorded the associated effort, barriers, and benefits. LFG supported this effort with access to technical support at no cost to the hospital.
Results: The results indicated 12 people dedicated 117 hours which was in addition to the project manager’s time and the majority of those people were department heads. Multiple LFG standards do not align perfectly with other accrediting bodies, meaning that specific changes would need to be made in order to score highly on several sections. The survey’s Section 2 required a disproportionately high number of hours due to the 40 hours needed for medicine reconciliation assessment. The CEO estimated that fully committing to achieving the LFG standards year-round would require $325,000 to fund additional nurse FTEs. The processes used to complete the survey and barriers encountered are included. Of particular interest for administrators, is how the volunteer survey is connected to LFG’s other posted hospital rating, the hospital safety grade which is calculated with or without survey participation.
Conclusion: The process takes significant time from leadership and committing to achieving the LFG standards would require significant funding. The immediate benefit was that departments used the survey to identify easy-to-achieve, 90-day quality assurance improvement goals. The CEO did not submit this first survey response, instead embracing it as preparation for possible subsequent participation.
Introduction
The Leapfrog Group (LFG), a Washington DC based non-profit, seeks to improve consumer utilization of hospital quality data in consumer decisions by providing easy-to-interpret online ratings for consumers (available at leapfroggroup.org). In this study the authors focused on two hospital-level reports: (1) Hospital Safety Grades which rate hospitals as A, B, C, D, or F (search hospitals at hospitalsafetygrade.org) and (2) Hospital Survey (survey) which is an assessment of a hospital’s achievement toward LFG-defined, evidence-based goals based (search hospitals at ratings.leapfroggroup.org). The survey is the heart of what LFG has pioneered in the consumer quality space; it is novel because it evaluates hospital protocols and processes and not just outcomes.1 At a high level, the survey asks hospitals to answer questions about their processes and protocols and then scores responses on 4 levels of achievement to the LFG standard (limited achievement, some achievement, considerable achievement, achievement of standard). If hospitals do not participate, their posted achievement status is “declined to submit.” While LFG will compute a safety grade without a voluntary submission, in those cases the grade is less reliable for consumer comparisons because LFG uses imputed data for missing survey information or eliminates the item. In short, both of these consumer reports rely on hospital voluntary submission.
The problem is many hospitals choose not to complete the survey.2 In Oklahoma, of the 90 hospitals,3 only 13 participate in the survey and nearly all are in the northeast corner of the state, leaving the majority of the Oklahoma consumers without access to survey information.
Because Oklahoma State University Center for Health Systems Innovation (CHSI) is dedicated to increasing access to quality care in Oklahoma, the authors wondered what LFG was asking hospitals to do to better understand our low participation and to inform innovative solutions to boost reporting. The authors could find no information on what participation entails or guidance for administrators to reference on ‘what it takes’ for a hospital to undertake a survey submission – an initial submission, in particular. While LFG does not charge hospitals for survey participation or rating,4 hospital administrators face the decision on how to dedicate hospital resources which already includes completing required reporting to multiple accrediting bodies.
Therefore, the primary aim of this study was to understand the effort associated with completing the survey for the first-time including the personnel, project management, and time. To achieve this aim, in 2023 the authors recruited a mid-sized, urban-based hospital without prior survey participation, hired a project manager, cataloged the process, measured effort in time, recorded barriers and estimated a subsequent survey effort. LFG provided some technical support and guidance to the project manager without charge.
Because through the course of the investigation we came to realize that the hospital chief executive officer’s (CEOs) motivation to participate in the survey was to affect the hospital’s LFG safety grade, an effort was made to understand how the survey affects the safety grade. Surprisingly, this connectivity is not overtly communicated by LFG in survey material or on their website. Therefore, a secondary aim of this study arose: to understand how the safety grade and survey responses are related. In this manuscript the authors describe the connectivity discovered and what factors were left unknown.
The motivation to affect the safety grade is understandable considering that LFG’s safety grade is its most promoted and recognized product. Hospitals often use this program to compare and promote themselves against competitors in their own markets,5 and grades are used by the media to compare hospitals.6 Moreover, because the long-term aim was to increase survey participation, it was critical to understand the survey connectivity to the safety grade to assess its potential to motivate safety-grade focused hospital administrators to submit the LFG survey.
Method
Participants
Project Management and Research. Oklahoma State University Center for Health Systems Innovation (CHSI) is dedicated to discovering innovative ways to decrease health disparities. CHSI partners with The Oklahoma Business Collective on Health (The Collective) a coalition of self-insured employer health insurance purchasers whose director serves as an LFG Regional Leader. The CHSI director and The Collective director recruited a hospital to undertake the survey with the support of a CHSI-funded 0.25 FTE project manager to organize the survey response. The project manager was a doctoral student in Oklahoma State University’s Health Care Administration program. CHSI oversaw project management and the investigation.
A 199-bed hospital located in Oklahoma was recruited. It is accredited by the Accreditation Commission for Health Care. The hospital gathered a team of administrators, quality improvement department staff, and department heads to tackle the survey.
This hospital had never participated in the LFG survey. At the time the project started, LFG had given the hospital a safety grade rating of “C.” The hospital CEO agreed to the project because he was interested in the survey as a means to raise the publicly posted LFG safety grade. He wanted to investigate how completing and submitting the survey might positively impact their Safety Grade.
LFG agreed to provide technical support to the project manager and assigned a key contact. CHSI communicated with the contact via email and phone calls and received response to questions regarding the survey reporting process and interpreting survey question intent. LFG’s stated mission is to drive giant leaps forward in the quality and safety of American health care. They aim to achieve these leaps by incenting safety and quality improvements through publicly available, consumer-friendly safety reporting.
Investigation
From April 1 to June 30, 2023, a project manager conducted the field study to collect the experiences of completing the LFG survey. The project manager collaborated with the hospital CEO and staff to record the process steps, record barriers, benefits, and measure their effort in aggregated time required for a staff to complete each section and the survey in aggregate, estimate second attempt/future effort changes and estimate with the CEO the potential costs associated with committing to LFG standards year-round.
Results
The project manager led the survey-response effort. Based on the content of the survey sections, the hospital assembled a 12-person survey response team which was comprised of four persons from the C-suite (CEO, Chief Nursing Officer, Chief Information Officer, and executive assistant), five department directors (Director Revenue Cycle, Director Pharmacy, Director Surgical Services, Director Women’s & Children’s Services, and Director of Quality) and three coordinators (Quality Improvement Coordinator, Medical Staff Coordinator and Manager Infrastructure and Support). The first observation was that the survey required effort by high-level leadership.
Each survey section was completed, submitted, and entered within the allotted timeframe as directed by the hospital CEO, prior to the submission deadline as required for attesting to accuracy and completeness. At the end of the survey period, the CEO determined he was not ready to submit the hospital survey answers to LFG and instead chose to embrace the first experience as a preparatory step for potential future survey submission. Many departments used the experience to define their 90-day Quality Assurance Performance Improvement goals. Participation generated new IT reports for expanded data collection and analysis. The following are results of the process used, effort tabulations, and future participation considerations.
Process Steps
The project manager recorded the process of motivating the staff and successfully collecting almost all section answers (Figure 1, Table 1). Included in the Appendix are communications utilized (i.e., Introduction email, workload introduction email and CEO debriefing report). Table 2 summarizes general barriers encountered.
Transparency in Hospital Safety: A field study of 'what it takes'for hospitals to complete the Leapfrog Group Hospital Survey
Effort
The combined effort to complete the survey was 177 staff hours, not including the project manager’s 0.25 FTE from April through June or technical support from LFG. The amount of time per section is represented in Figure 2. Section 2 required a disproportional amount of effort, more than double the time required by the Quality Department to help coordinate the entire response effort. Section 2 took 30 hours of the Directors time overall and 40 hours of a non-manager’s time for the medicine reconciliation assessment. In addition, the computerized provider order entry (CPOE) portion required multiple departments and levels to participate including a physician, manager, and non-manager role. The portion is timed, requiring multiple people, lasting six hours (3 hours elapsed time to complete Steps 1-2, then 3 hours elapsed time to complete Steps 3-6). Moreover, Section 2 takes more time due to the need to coordinate other leader schedules to complete the CPOE portion. Section 2 involved 78 total hours of time (Figure 2).
Survey and Safety Grade Connectivity
Because the authors were repeatedly asked, “How are the survey and safety grade connected?” The question was posed to LFG support. LFG replied to that inquiry that the survey and safety grade are two distinct measures; however, the authors discovered that they are not completely independent. As mentioned in the introduction, explanations of how these two separate programs are related were not readily available on the LFG website nor did the survey indicate what measures were used by the safety grade. Researchers sourced research articles and read the safety grade methodology report buried on the LFG website.7 Of note, none of these are written to LFG’s targeted consumer reading level (6th grade).
The largest takeaway for administrators is understanding that the safety grade is calculated from publicly available CMS data and data gathered through the hospital’s voluntary survey submission. So, it is feasible that survey submissions affect safety grades. However, because LFG does not publicize cut-off points for safety grade performance, where the potential lies is not completely clear.
Based on the researcher’s investigation (not provided by LFG), the authors describe what could be found on how survey answers affect the safety grade. First, it is necessary to introduce the safety grade. The safety grade is calculated by formulas developed by LFG. Formula development was published in the Journal of Patient Safety in 20138 and is occasionally updated.7 The final safety grade, an A-F score, is a 50-50 composite of hard data (i.e. infection outcomes) and soft data (i.e., safety process assessment). Interestingly, that 50-50 split was admittedly “arbitrary” as stated by the authors in the original 2013 publication regarding safety grade calculation. Although there are 2 sections to the grade, no individual grades are calculated for outcome and process independently. Only one 50-50 composite grade is reported to consumers.
The authors contend that the mixing of the two types of variables is a credible threat to the validity and utility of the safety grade. The hard data (hospital infection scores) are calculated using 100% (10 of 10) of publicly available data from Centers for Medicare and Medicaid Services (CMS); whereas process scores are calculated with only 42% (5 of 12) of publicly available CMS data. Over half of the process scores are dependent upon survey participation.
In the absence of survey data, process scores for an individual hospital are either: eliminated and then its weight shifted to other measures, calculated using imputed data, or calculated using other CMS data. LFG imputes data by using the lowest mean scores of the hospitals’ cohort data determined by whether the hospital is rural/urban, safety-net/not safety-net, and teaching/ non-teaching. For the three measures where no cohort data is available to impute, the weight of these three measures are shifted to other measures, although how much and to which variable it is shifted is not indicated. Without survey participation, the LFG uses a hospital’s CMS cost report to determine compliance with ICU intensivists’ staffing levels, a controversial standard whose validity has been argued for over two decades.9 Precisely which scores get imputed versus eliminated (and formula weight shifted) are presented in Table 3.
In summary, the authors were not able to locate via LFG technical assistance or LFG publication, a clear explanation of:
• exactly how the weighting of process measures within the safety grade changes with/without survey participation,
• the degree to which cohort measures end up representing a hospital’s process score when LFG uses them in replacement of survey participation,
• the degree to which cohort process scores affect final grades, or
• any testimonial from any similar cohort hospital explaining what they learned about responding to the survey and its effect on their safety score.
Future Participation
In order to weigh future participation in the study, the authors collected three factors: (1) post-survey sentiments around validity of survey (Table 4); (2) estimated effort for next survey completion and (3) estimated cost to committing to the LFG survey achievement standards.
Future Effort Estimations
The staff anticipated that their next survey attempt would require 7% less effort. Confidence in that time metric was somewhat low due to the unknowns of annual changes to the survey and effects from high intensity standards such as handwashing observations. Additionally, Section 1C declined to report an anticipated change. Notably the most work-intensive section (Section 2) only estimated an eight hour decrease in effort. Estimations by survey section as well as staff explanations for estimation are reported in Table 5.
Future Participation FTE Estimation
The CEO estimated that a commitment to maintaining the LFG standards would require hiring at least 4.0 FTE personnel. In the hospital team’s opinion, the level of hand hygiene observation and real time teaching 24/7 requires the greatest FTE time of all of the individual LFG requirements. Due to the clinical and technical nature of the education required, the likely hire is a nurse, or nurse adjacent positions. Based on an average hourly rate of $38 plus benefits, 4.0 FTE translates to a $325,000 commitment, not counting the current team’s staff hours for completing the survey.
CEO and Staff Sentiment
While not formally collected, because the aim of the study was to understand ‘what it took’ to complete the survey, the authors are compelled to describe the experience of the staff sentiment throughout the process as managing these attitudes and questions are central to leading a first-attempt survey submission.
The frontline section leaders expressed a persistent lack of clarity as to the demonstrated value (e.g., commercial, consumer, predictive of future performance). Staff regularly provide metrics to hospital oversite organizations and did not understand how LFG survey participation addressed consumer concerns. The authors posed these questions to the LFG who responded to these inquiries with general information about its use by consumers and employers. However, the staff questioned the data used in the local market as examples of local use of data or local testimonial for the value of the survey were missing.
Staff were frustrated by LFG standards that often exceeded accrediting bodies’ and it seemed counterproductive to self-report a low achievement when adjustments could be made for higher achievement in subsequent survey responses.
The hospital CEO embraced the strategy for the first attempt to act as a preparation step for potential subsequent submissions. One stated reason to trial the survey was because the time between the survey download in April to the upload deadline in June, the timeframe of this investigation, was too tight to make even easy-to-achieve process adjustments to improve scores by more closely aligning with LFG standards.
The project manager and the CHSI team found that LFG maintains a number of websites and there are many links between and within them. Finding assistance and clarifications online was difficult. The option of payment for watching informational webinars was explored. However, the CEO chose not to since there was a project manager assigned and the webinar advertisement stated that all the same information was provided in the written instructions. The webpages were difficult to navigate; even finding the two ratings for hospitals was difficult for the authors, requiring clicking between several pages. There is no one place where a site visitor can search a hospital and then choose/click which report(s) to view.
Discussion
In an attempt to answer the question, what does it take for a mid-sized hospital to make a first attempt to respond to the LFG survey, the authors concluded that it takes significant time from the executive suite and administrative staff and there are multiple barriers that may lead hospitals to abandon the effort, not the least of which is the survey size. The study team was able to mitigate those barriers by hiring a dedicated project manager, dividing the work, and organizing IT data requests. The project manager believed the decision to gain initial cooperation from frontline managers by carefully dividing the survey into small, feasible sections was most vital to her success. Staff predicted that the effort required would be somewhat less in subsequent attempts.
The authors believe downloading a survey which is free in preparation for a later submission is a good potential strategy for first-time submitters that would result in higher participation and CEO confidence in the submitted survey accurately representing their hospital’s safety level. Submitting the survey is one decision - a more complex one is if the hospital wants to take the next step to committing to achieving LFG standards. For the hospital in this study, the CEO estimated that was a $325,000 per year commitment, a cost the CEO expressed could not be justified given their current levels of reporting to multiple other quality agencies and prioritization of those accrediting body standards over LFG.
First-time submitters should know that this study found the LFG webpages and online resources difficult to navigate and use which made the dedication of a project manager for the LFG survey even more important.
The authors further concluded that administrators lack clear information about the connectivity of the two LFG reports (voluntary survey, nonvoluntary safety grade) because LFG does not publicly provide a crosswalk, briefs, or testimonials on this. It is hoped that this study’s description of the connectivity helps to fill that gap and motivates LFG to communicate the connectivity clearly. It is the authors’ opinion that overlooking this relatedness and not clearly describing the opportunity to CEOs is a missed opportunity to motivate other administrators to participate in the survey and to generate more information for consumers. LFG should clearly report what changes in the safety grade calculation when a hospital completes the survey and even allow CEOs to trial how their submission would affect their grade.
A strength of this investigation is that it represented first-time impressions of work done by a hospital and project team who had never completed a LFG Hospital Safety, Quality and Resource Use Survey prior to this study. However, the authors’ inexperience also provides a limitation. While every effort was made to locate and capitalize on the instructional materials and grading information LFG provides, the authors acknowledge that it is possible some were missed some. The most significant limitation of the results is that they are from a single site and team. The authors acknowledge effort will vary between hospitals, especially based on their access to report-building staff and data analysts – without these, the effort required would greatly increase. The value of this investigation for administrators is not the discovery generalizable knowledge but rather a transparent reporting about “what it took” for a hospital to complete the survey for the first time given the current lack of information.
References
1. Moran J, Scanlon D. Slow progress on meeting hospital safety standards: learning from the Leapfrog Group’s efforts. J Patient Saf. 2013;32(1).
2. Giant Leaps for Patient Safety: The Leapfrog Group 2021-2022 Annual Report. Accessed December 28, 2023. https://www.leapfroggroup.org/sites/default/files/Files/LEAPFROG%20GROUP%20AR.pdf
3. American Hospital Directory. Accessed December 28, 2023. ahd.com
4. The Leapfrog Group. Survey Login and Materials. Accessed December 28, 2023. https://www.leapfroggroup.org/survey-materials/survey-login-and-materials
5. How are “A” Hospitals Sharing Their News? Accessed December 28, 2023. https://www.leapfroglicensing.org/sample-promotions.html
6. Arougheti I. Three Illinois hospitals keep their A streak alive in Leapfrog hospital safety grades. Chicago Tribune. November 6, 2023. https://www.chicagotribune.com/business/ct-biz-chicago-illinois-hospitals-earn-leapfrog-safety-grades-fall-2023-20231106-nwcgta4rhfgklmrwqalzj24p6q-story.html
7. Leapfrog Hospital Safety Grade Scoring Methodology Fall 2023 Safety Grade. Accessed December 28, 2023. https://www.hospitalsafetygrade.org/media/file/Safety-Grade-Methodology-Fall2023.pdf
8. Austin M. Safety in Numbers: The Development of Leapfrog’s Composite Patient Safety Score for US Hospitals. Safety in Numbers.’. The Johns Hopkins University School of Medicine. 2013;9:750.
9. Manthous CA. Leapfrog and critical care: evidence- and reality-based intensive care for the 21st century. The American Journal of Medicine. 2004;116(3):188-193. doi:10.1016/j.amjmed.2003.08.032
Acknowledgements
Oklahoma State University Center for Health Systems Innovation (CHSI) thanks the hospital executives, department heads, quality improvement department and all staff who assisted us in gathering our data and made an earnest attempt to complete the Leapfrog Group’s hospital safety survey. The authors appreciate the CHSI executive director, William Paiva, for agreeing to fund the research project and the graduate assistant to make this project a reality. Additionally, the Oklahoma State University Health Care Administration program administrators helped to identify and recruit a qualified project manager among their doctoral students. Finally, CHSI thanks the Leapfrog Group for the enthusiasm for our research, interest in their own process improvement possibilities, for providing technical assistance without charge, and not the least of which leading a national, transformational idea – making health care consumers informed about their health care choices.
APPENDIX: Project Manager Communications (Intro Email, Workload Intro Email, CEO Debriefing)
