Designing for Impact: An Approach to US Based Executive Education for Healthcare Professionals from China

: There is limited evidence on how healthcare executive education programs can be structured to achieve impact, particularly within the rapidly advancing healthcare systems in China. This study reviews the design and evolution of three programs hosted by the University of Michigan Medical School to engage mid-level healthcare leaders from three healthcare institutions in China. Program participants included 40 Chinese physicians and administrators from 13 hospitals across the three healthcare institutions. The Kirkpatrick model was used to structure an approach to evaluate the learner outcomes. The programs were well received, with the effectiveness score in the first three Kirkpatrick-levels of reaction, knowledge acquisition and application of learning being 4.61, 4.34, and 3.55, respectively (on a five-point Likert rating with 5 as the highest rating). The results demonstrate the ability to co-design executive education programs with learners who advance the expected outcomes beyond mere satisfaction with their participation in the program. This approach is increasing the demand among healthcare institutions and their employees in China for these programs at University of Michigan Medical School.


Introduction
Healthcare leaders must be equipped to address ever-changing challenges related to quality, affordability and accessibility, making the need for strong leadership training and competencies critical to their success and that of their organizations [1,2].
China's healthcare system has the added complexity of recent large scale, rapid healthcare reform. In 2009, the Chinese government set out to establish an equitable and effective health system for all citizens by 2020 [3], which included the goal of providing universal health coverage [4].
A national health services survey [5] indicated that more than 30% of patients were unsatisfied with inpatient care, mainly because of process inefficiency, concerns about quality, safety, and the attitudes of medical staff [6]. In reaction to this situation, the central government prioritized developing the skills of healthcare providers [7]. This included investing 30% of government funding in improving infrastructure and training primary healthcare providers [8].
With the heightened emphasis on improving the quality and efficiency of healthcare delivery especially in tertiary level hospitals, Chinese healthcare institutions have been challenged to consider and utilize innovative approaches to address these issues. Training of Chinese healthcare professionals on how to efficiently and effectively manage healthcare systems is becoming increasingly urgent.
While many healthcare organizations in the US have offered healthcare leadership development programs, important gaps in understanding the impact of such programs continue to exist. Available literature [9,10] shows little evidence that such programs exert significant personal or organizational impact. Similarly, after surveying faculty development/affairs deans of the 161 Association of American Medical Colleges member schools, Lucas, Goldman, Scott, and Dandar [11] found that evaluation beyond participant satisfaction was uncommon. Specifically, only 38% assessed post-program learning, and 30% assessed post-program behavior change among participants.
To address the knowledge gap on the impact of healthcare leadership development programs, as well as to explore using a participatory co-design approach with stakeholders, the authors co-developed and evaluated three pilot executive education programs for Chinese healthcare professionals.

Settings and Participants
Many institutions in the US have started to provide executive education programs for healthcare professionals from China. Cleveland Clinic [12], Harvard Business School [13], and Columbia University [14] are a few examples. However, these programs are often directed at more senior Chinese executives who have the ability to travel and afford the costs. This leads the needs of mid-level managers clearly not addressed.
In 2017, two authors (AH, JCK) of this study implemented a pilot executive education program at the University of Michigan Medical School (UMMS) in conjunction with the university-owned hospital and health system (called Michigan Medicine) in Ann Arbor, Michigan. The program was three weeks in duration, with 15 participants from Peking University Third Hospital (PUTH). Designed for mid-level managers with a focus on hospital management and leadership development, the curriculum consisted of lectures and hands-on experiences in Michigan Medicine hospital settings. The program was well-received and subsequent strong interest was expressed to continue the training.
In early 2019, the Executive Education Program office was established. In the same year, the authors designed and implemented three executive education programs, which are the basis for this report. The participants came from 13 hospitals in China and were organized by three Chinese healthcare institutions whose leadership had been engaging with UMMS on other academic collaborations (e.g. research projects reported by Kolars et al. [15], and therefore chose to offer their physicians and administrators the opportunity to participate in an executive education training program at UMMS. The three institutions were PUTH (N=16), Shenzhen Medical Association (SMA, N=9), and Xiangya School of Medicine at Central South University (CSU-XSM, N=15). The participants were cohort-based (i.e. coming together as a team from one institution at a time), and included physicians, faculty, and administrators from clinical and basic science, as well as departments such as human resources, clinical affairs, finances, etc. The executive education training programs were organized by the unit called Global Research, Education, and Collaboration for Health (i.e. Global REACH) within the Deans Office of UMMS in May, July, and September 2019.

Contextual Framework for the Design
In an effort to develop a healthcare leadership program that is relevant and useful, we engaged the stakeholders during the design process. Using Kern, Thomas, and Hughes' [16] six-step approach, we created the curriculum for teaching various healthcare leadership concepts. This enabled us to provide "a practical, theoretically sound approach to developing, implementing, evaluating and continually improving educational experiences …". Kern, Thomas, and Hughes' framework for curriculum development has successfully been applied widely within medical education across multiple specialties and training [16][17][18]. The proven efficacy and versatility of this framework made it an appropriate foundation and applicable for the design of these programs. The effects of the design process in particular through the involvement of stakeholders were well documented [19][20][21]. Having stakeholders working collaboratively can eventually facilitate ideas being adopted [22,23]. We speculated that introduction to Kern, Thomas, and Hughes' approach would increase the collaborative effort in curriculum development, and that application of the approach would lead to a customized educational product for the participants.

Problem Identification and General Needs Assessment
Problem identification and needs assessment provided an opportunity to better understand China's healthcare environment. It included a review of the relevant published literature and other available information (e.g. national health services survey [5]).

Targeted Needs Assessment
Targeted needs assessment with stakeholders at each institution formed the basis of the curriculum, grounding the curriculum in the specific needs of the learner cohort rather than the needs of learners in general.

Goals and Objectives
The original purpose for exploring executive education programs with institutions in China was to create mutual awareness and advance collaborations between UMMS faculty and those in China for shared projects, research, and innovations. The goals of the training program were to improve the participants' knowledge and skills in performing their leadership roles and achieve better patient-centered care and process efficiency.
At the early stage, two important objectives were identified. The first was to ensure that the complexity of each healthcare institution was fully appreciated, and that program content was sensitive to this context. The second was for the UMMS design team to gain credibility with senior executives of the institutions to ensure a training program could be produced that would be viewed as valuable.

Educational Strategies
Design of the curriculum and learning experience was lead at UMMS by two of the authors (AH, JCK) who have several decades of experience and familiarity with the Chinese healthcare system. In China, separate design teams worked with them from the respective institutions (i.e. the president from PUTH, director from SMA, and dean of education from CSU-XSM) ensuring early buy-in from key constituents of the program. It is worth noting that this process took place iteratively through multiple face-to-face as well as virtual meetings between the UMMS and the institutional design teams in China. The design team researched the executive education program offerings from several comparable healthcare executive education programs for health-care leaders from China at peer institutions [12, 13] and benefited from the executive education expertise at University of Michigan's Ross School of Business [24].
The curriculum (see 2. 3) was composed of formal classroom learning, targeted individual shadowing opportunities (upon request by the respective institution), and intensive team dynamic exercises. Between formal lectures were multiple opportunities for tours of the patient care and research facilities and shadowing of staff and faculty. A customized leadership challenge program, consisting of a full-day outdoor program with interactive activities focusing on team building, communication, and trust was also included.
The program was open to requests for customization during the design and implementation stages, which included content modifications, shortening or lengthening the program, or any other adjustments based on needs and to accommodate group sizes.

Implementation
Target learners for the curriculum were mid-level Chinese healthcare professionals selected by the leadership of the three Chinese institutions. The curriculum began with potential participants completing a pre-program questionnaire to assess their baseline knowledge and skills surrounding healthcare leadership skills. Two to three weeks before the program start date, participants were provided pre-reading materials relevant to program lectures. The amount of reading materials varied depending on the subject matter, with number of readings ranging from 1 to 11, and number of pages for the three programs being 223, 180, and 134, respectively. Participants were required to complete the reading at their own pace, and their progress was tracked by the program management team with the help of the group leader from the respective institution.

Evaluation and Feedback
Pre-program, daily and post-program (immediate and five months later) questionnaires were the primary evaluation method used. Based on a comprehensive literature review of previous studies on evaluation, the Kirkpatrick's evaluation model was used to guide the evaluation process. Our evaluation consisted of 18 questionnaires (see 2.4).

Curriculum
Based on Kern, Thomas, and Hughes' [16] six-step approach, the major building blocks of the curriculum emerged as revolving around the continuum of general and specific needs assessment, content development, delivery, and evaluation.
All the cohorts shared a common core of materials, but each of the three cohorts focused on one track depending on their institutional needs. Specific curricular elements were identified for each track.

Core Curricula
The education curricula and the learning objectives of the core curricula can be seen from Table 2.

Track-Specific Curricula
Three tracks were identified: Hospital Management and Leadership, Clinical and Translational Research, and Medical Education.   Each program took place during a three-week on-site visit to the US. Over the course of the three weeks, each participant engaged in 112 hours of programmatic learning, including lectures, tours of patient care facilities, shadowing opportunities with faculty and staff, and an intensive customized team dynamic exercise. Program presenters with relevant expertise were drawn from the staff and faculty of the University of Michigan Health System and numbered 118, 52, and 53, respectively for each three-week program. The significantly higher number of staff and faculty involvement was due to the request of adding targeted individual shadowing. Upon completion of the three-week program, each participant was required to prepare a ten-minute PowerPoint presentation to the UMMS Global REACH leadership team addressing three key take-away concepts or models from the program that they found particularly useful. In addition, they outlined three plans of work that they intended to pursue after returning to their home institutions. Specific attention was placed on describing projects that addressed a process, issue or problem that could be improved along with a plan, resources needed, and a timeline for implementation. Finally, participants were required to pass a final exam before receiving a training certificate.

Evaluation Model
The programs were evaluated using the Kirkpatrick model [25] which assesses the effectiveness of training programs at four levels: (1) reaction to the training experience; (2) learning outcomes and increases in knowledge, skill, and attitude towards the experience (how much attendees learned the content); (3) participants' change in behavior and improvement (whether the learning transferred into practice in the workplace); and (4) results (the ultimate impact of Healthcare Professionals from China training). Our evaluation used levels 1, 2, and 3 of the Kirkpatrick model. Level 4 was not utilized for this evaluation due to resource constraints related to long-term follow-up, but approaches to incorporate level 4 into future program evaluation will be examined.
The Kirkpatrick model provides a framework for evaluating the effectiveness of training for any industry including healthcare, and it allows for measurement of a limited number of variables, ease of evaluation criteria, and lack of need to collect the basic data and learners' previous performance [26].

Questionnaires
All participants completed 18 questionnaires: one pre-program questionnaire, 15 end-of-day questionnaires, one post-program questionnaire on immediate completion of the program, and one follow-up questionnaire five months later. The questionnaires were established by the authors with input from stakeholders in China and included open-ended questions for qualitative and quantitative information. Pen-and-paper questionnaires were used for the first two cohorts, and web-based questionnaires were administered for the third cohort.

i) Pre-Program Questionnaire
The pre-program questionnaire (Table 6) consisted of 33 items and established the participants' baseline knowledge and which particular curricula elements they were most interested in learning.

ii) End of Day Questionnaires
These questionnaires were administered at the end of each program day to monitor satisfaction and identify improvement opportunities. They were reviewed daily so that real time adjustments could be made on both program content and logistics. The evaluation was organized into four areas: topics, learning experience, tour experience, and speakers.  (Table 7) were similar to the pre-program questionnaire, except that the questions were geared towards the end of program assessment. Comparison of the pre-program and post-program questionnaires allowed for assessment of overall program satisfaction as well as acquisition of knowledge and skills.

iv) Follow-up Questionnaire Five Months after Program Completion
A five-month post-program follow-up ( Table 8) provided evaluation of the program impact to see whether knowledge and skills were applied and passed along to other individuals within the institution.

Final Exam
To assess learning, a final exam was administered on the last day of the training. The authors created the final exam based on the main topics and contents covered in the training. The exam was organized into three types of questions (true/false, short answers, and essay questions). Depending on the tracks, number of items in the exam varied between 20-25. Examples of short-answer included asking participants to define or explain these terms in three sentences or less: high reliability care, Lean Healthcare, competency-based assessment in medical education, and clinical trial protocols. Essay-question items included questions that required the application of content knowledge, for example, Explain Waste in regard to Lean Thinking. Where do you find waste in your own hospital? Participants were provided one-hour of class time to complete the open-book written exam individually using pen-paper. The exam was graded on a percentage-based system.

Data Analysis
The authors used descriptive and inferential t-test to examine the numerical data and thematic analysis for the responses from the open-ended questions.

Results
A total of 40 physicians and administrators participated in the evaluation which made the response rate to be 100%. Table 9 shows the characteristics of participants from the 13 hospitals. The top two categories of participant specialty department were internal medicine (8, 20%) and pediatrics (6, 15%); in terms of healthcare roles, physician (18, 45%) and administrators (14, 35%) were the top two categories.

Kirkpatrick Level 1: End of Day Questionnaire
The high level of satisfaction (Table 10) indicated that the overall training experience was positive, and that the participants considered the training topics and activities relevant, instructors knowledgeable, with contents that are practical and applicable to their own environment.

Kirkpatrick Level 2: Pre-Post Program Questionnaire, Knowledge Acquisition
Comparing results from pre-and post-program questionnaires, participants perceived significant increases in knowledge and skills, as shown in Table 11. Performance on the final exam (Table 12) also reflects a good understanding of the learning materials.   All three cohorts use the same scale: 1=Strongly disagree, 2=Somewhat disagree, 3=Unsure, 4=Somewhat agree, 5=Strongly agree.

Kirkpatrick Level 3: Follow-up Questionnaire Five Months after Program Completion
The result of the Part 1 general evaluation (Table 13) indicated that the participants applied the learning within their institutions. However, some participants reported being slightly unsure about applicability of learned concepts upon returning to their home institutions.
Responses from the open-ended questions shed light on some of the unsure rating. These responses were thematically analyzed. We report the overall themes and subthemes developed, and provide representative comments.

i) Common Challenges in Learning Application upon Return to China
One of the common themes that emerged from the responses is the lack of bandwidth and resources to be able to incorporate learning upon returning to home institutions. While this applied to all three institutions, the participant number that reported on this varied. Only two participants from PUTH (N=16) and CSU-XSM (N=15) felt that lack of resources was a contributing factor. One participant from PUTH explained that even though the management concepts and tools she learned from the training helped her solve "some of the existing problems in the training of residency and graduate students", "too much clinical work, too little time …" contributed to her inability to use new learning. The other participant reported that "Everyone is busy with work and the discussions were not extensive". The two participants from CSU-XSM reported similar issues.
However, the participant number that reported on lack of resources was higher for SMA (N=9). Four participants indicated that lack of resources was the reason that new learning was not adopted at work. They reported on "heavy workload", "no time to apply for a grant", "not having enough knowledge how to implement the projects", "no decision-making power due to lack of seniority", and "not enough funding". One junior physician commented that her "seniority and title (was) not enough to implement research projects", and "Young doctors do not know how to start clinical research". All these factors made the application of new learning infeasible at SMA.

ii) Application of New Learning Appears to Improve Job Performance at PUTH and CSU-XSM
The job performance results for SMA were not as apparent as those for PUTH and CSU-XSM. Twelve of the 16 respondents from PUTH reported quality improvement in patient services, as a result of applying the concepts of A3 tools (i.e. a structured problem-solving and continuous-improvement approach) and huddles (i.e. a short meeting in a clinical setting at the start of a day to discuss patient concerns, safety issues, and updates) learned from the lean healthcare training. Specifically, three participants reported that door-to-needle time (for the acute managements of life-threatening conditions requiring vascular injection) was shortened dramatically. Other areas of reported improvements included: smooth flow and increased numbers of patient appointments; timely administration of medications within 60 minutes; process improvement of neonatal critical care and obstetric and resuscitation unit as well as premature infant care; increased interest of residents and undergraduates in laparoscopic skills and in surgery; standardization of specimen collection and retention; improvements in the management system of residents and graduate students; planning and building a new clinical simulation center for medical education enhancement; and lastly, implementation of performance reform project. Reported impacts also included perceived enhancements in problem solving, leadership, flexibility, dependability, and a deep sense of ownership.
The application of learning for CSU-XSM was more centered around curriculum reform, clinical teaching method, and evaluation system. Six of the 15 participants reported progress on reforming the current curriculum based on the competency-based University of Michigan model. Eight participants reported on the process of integrating clinical teaching methods and techniques from their learning, especially on how to increase student engagement in large lecture halls. The other areas of application of new learning included MOOC (Massive Open Online Courses), PDSA (Plan, Do, Study, and Act) in the improvement of teaching, and how to balance online and offline learning. It is worth noting that the CSU-XSM participants described unique challenges related to a COVID-19 pandemic just starting to unfold which provided some limitations to implementation of plans.
iii) Quality and Process Improvement The most dominant theme and a clear attribute of the training was the application of the lean management and A3 concepts to improve workflow, accuracy, and safety. One participant described his experience with using the visual management tool -a component of A3 that allows clear visualization of workflow and project planning: "In my opinion, the visual management tool A3 is quite effective for multi-person collaboration and long-term advancement work. Relevant personnel often stand in front of the board to check the progress and personal tasks. The phenomenon of 'too long ago, can't remember, and nothing to be done' no longer exists. Through the demonstration of our department, other departments have also come to visit and learn." Eighteen of the 40 respondents cited work improvement as a result of applying learning. One participant summarized many of the changes effected by the quality and process improvement skills that she acquired as having: "improved the accuracy of nurse triage, enhanced lab efficiency, reshaped lab, (made) more space for discussion, established an open and timely communication system in the ophthalmology lab, (and) reorganized the sequence of lab testing to be more logical and efficient." As a result of the application in this area, one participant from PUTH reported that the time for patients waiting for blood transfusion was reduced from 110 minutes to 80 minutes. After more measures were implemented, the time of blood transfusion was shortened to 10 minutes.
With the program satisfaction and positive impact, the three institutions were eager to sign on for future trainings. SMA signed a multi-year agreement to secure at least one training session each year. PUTH and CSU-XSM also reached verbal agreements with UMMS to send 15 trainees each year to continue the program. In addition to the three partner institutions, three other Chinese healthcare institutions expressed strong interest in participating in future training. We were also approached by several private Chinese healthcare consulting firms interested in collaborating with UMMS to deliver educational programs to healthcare executives.

Discussion
We sought to create a healthcare leadership program for participants from China that stood apart from other offerings in how it was designed and evaluated. This program, co-created by stakeholders from China and UMMS was customized for participants from three different institutions. One of the intents, to foster collaboration between UMMS and the institutions, will take more time to realize. In this report, we present the co-design process, tailored programming, and an approach to measuring outcomes. The results indicate that participants were very satisfied with the program with improvements in knowledge and skills. Furthermore, they were oriented upon return to their work environments to applying what they acquired in the program to improve performance in their own healthcare contexts. The actual results of these improvements on patient care or system performance (i.e. Kirkpatrick level 4) is the ultimate metric of impact for training programs but is beyond the scope of this report.
Our study addresses a few important gaps in the literature on leadership training: 1) there are few available studies exploring co-design with stakeholders [27,28], and 2) few programs report rigorous evaluation of training impact [9,10,29].
The participatory co-design process led to mutual understanding and learning between the institutions, which is one of the extensively documented impacts of co-design [30]. Our design process engaged all levels of the partner institutions, from top executives in Chinese institutions to the actual participants of the programs. The involvement of all levels of stakeholders enables the development of a multi-dimensional collective initiative, therefore laying the foundation of building personal relationships, trust, and creating a safe space for discussion about the needs of stakeholders. These effects of the co-design process were well Healthcare Professionals from China documented by previous studies [19][20][21].
The literature on participatory design also describes how having stakeholder engagement from the beginning eventually facilitate greater adoption of learning into practice, which has been attributed to greater feelings of ownership and increased logistical and financial support [22,23]. A number of participants were clearly able to implement their learning from the program back at their home institutions to create real change, which may have had something to do with buy-in during this process. One opportunity for improvement born from this data is the need to emphasize from the onset that attendance in this program may inspire participants to envision major changes in collaboration with their leadership. Our findings demonstrate that some participants did not feel that they were given the space or the opportunity to apply their new knowledge. We speculate that having this as more of an explicit goal from leadership of their institutions may result in better project implementation from the learners upon their return.
Another distinguishing aspect of our program was the evaluation of its effectiveness. A growing number of organizations are demanding executive education that has a proven positive impact. Surveys by the Financial Times/IE Business School Corporate Learning Alliance show that in 2018, 41% of executives were choosing a provider based partly on this criterion, up from 25% in 2016 [31]. This increase reflects cautious corporate spending as well as the intensifying competition in executive education, making the impact measurement especially important [31]. Evidence suggests that learning outcomes are heavily influenced by whether well-articulated objectives and metrics used to assess program performance are met [28]. Studies by Tushman, O'Reilly, Fenollosa, Kleinbaum, & McGrath [32] indicate that program impact should be measured through individual learning, individual behavioral change, organizational change and organizational results. In order to assess such outcomes, individuals should be questioned on a) their motivation for attending an executive education program, b) their preparation prior to attending, c) their application of learnings/concepts/methodologies since attending the program, d) the depth of their knowledge post-program, and e) the behavioral change/knowledge transfer that had occurred. Our evaluation was designed to better understand these aspects. Tushman, O'Reilly, Fenollosa, Kleinbaum, & McGrath [32] also found that organizational results would be considerably stronger when custom programs are collaboratively designed by all the stakeholders, utilize action-based projects and recruit program participants as in-tact teams.
One prominent limitation from the design of this program was that it focused on levels 1, 2, and 3 of the Kirkpatrick model, but did not include level 4. Many studies acknowledge the difficulty of evaluating the 4 th level [33][34][35] in terms of quantifiable factors such as patient welfare, waste reduction, or profit margin. Often these parameters are confounded by other variables and take more time to recognize. Evaluating this level was made more difficult by the fact that the 16 participants from PUTH came from three different hospitals, the nine participants organized through SMA were from five different hospitals, and the 15 participants through CSU-XSM came from five hospitals. However, with the success of the first cohort and the expectation to have more training programs in the near future, we will explore approaches to adding the level 4 evaluation to our program planning.
Furthermore, the results are based on small samples sizes due to physical space limitations in the clinical environments. However, as the program continues to grow and more data are generated, the findings of this study will be further substantiated.

Lessons Learned
The lessons learned and values realized from implementing the program to the first three cohorts apply both to our partner institutions and UMMS. Moreover, these lessons appear to be transferable to other customized executive education programs.
Commitment from the leadership at the participants' institutions to co-design the curriculum and support their implementation of knowledge upon return is critical to maximize the impact of the program. The leadership presence focuses the attention of program participants by making the latter aware of the personal and institutional significance of the learning opportunity. It also enables projects to be implemented and new learning to be reinforced upon returning to the home institution. Whereupon leadership support is lacking, the opportunity for growth is limited for those who wanted to bring real change by applying the knowledge and learning.
The continuity of the faculty especially for the core materials greatly increases program credibility. Utilizing a largely unchanged faculty pool allows their healthcare knowledge to continuously improve from program enhancements and the understanding of contextual applications to the complex challenges of the healthcare industry. Iterations of content delivery is as important as the content itself.
Similarly, the roles of the UMMS design team members play a pivotal role for partnership harmony. Their involvement is critical as they act as the point of contact for UMMS in maintaining the integrity and flexibility of the program. Encouraging involvement from program alumni may also have beneficial effects for future cohorts. It provides a positive reference source within the institution, a networking opportunity, and valuable support system for trends of project implementation within the institution.
Finally, it is essential to continuously upgrade and update program parameters and materials. Hence, the program design team that originally drafted the program content are continuously improving the quality of program content and delivery. They meet constantly to review and revise materials, striving to incorporate relevant new ideas from the healthcare industry.

Conclusion
Our study provides an example of successful co-design of an executive training program between three healthcare institutions in China and University of Michigan Medical School. It has underlined the importance of matching contextual healthcare needs with their instant application through action-based learning projects of personal and institutional significance. Our program represents a trusted partnership in the healthcare industry where substantial competitive challenges remain a constant. The impact of our program depends on retaining relevance, flexibility, and continuous improvement through the co-design process. Our findings suggest that healthcare executive education programs can successfully engage stakeholders during the design process for shared decision making, program buy-in, and the effective transfer of knowledge, skills, and information. We also found that these programs can and should be rigorously evaluated to understand impact and areas for improvement.