Construction of Quality Indicators of Outpatient Care Based on Delphi Method and Analytic Hierarchy Process

Objective: We aim to build a standard quality evaluation indicator system to provide a scientific quantitative evaluation tool for the assessment of the quality of outpatient care. Methods: we drew up the content of the indicator system for quality of outpatient care through searching literature at home and abroad and decided the indicators and their weight based on the Delphi method and analytic hierarchy process. Results: The collection rates of the expert consultation questionnaires were 95.45% and 100% for the first and second round and the coefficients of authority were 0.881 and 0.877, and Kendall coefficients of concordance were 0.303 and 0.313 respectively. The quality evaluation indicators of outpatient care consists of 3 first-level indicators, 14 second-level indicators and 56 third-level indicators. Conclusions: The indicator system for evaluation of quality of outpatient care is scientific and reliable and thus can be used as a tool for the evaluation of the quality of outpatient care, providing basis for continuous improvement of outpatient care quality.


Introduction
Outpatient department is seen as a window of the hospital and the quality of its nursing service has a direct influence on the patients' medical experience and hospital reputation. The outpatient department is characteristic of receiving more patients, treating various diseases and having complex visiting process. Besides, the nurses are insufficient in outpatient department. These may reduce the quality of the outpatient care [1,2]. In 2015, the National Health Commission put forward the "Plan for Improvement of Outpatient Care Service", which advocates improving quality of outpatient care service [3]. The indicator system for evaluation of the quality of nursing care as a quantitative tool for evaluation of nursing service has a guiding role in managing the quality of nursing care [4]. Currently, the quality indicators of clinical care are constructed for evaluation of quality of ward and specific nursing in Guangdong Province and even across the nation. There lacks a standard for evaluation of the quality of outpatient care. Hence, it is of significance to construct an indicator system for the assessment of the quality of outpatient care. The current study thus constructs a scientific indicator system for evaluation of the quality of outpatient care so as to provide a scientific basis for the construction and evaluation of the quality of outpatient care.

Building a Research Team
The research team consists of 7 members among which two members have senior title, one has deputy senior title, two have intermediate title, and two are graduate students majoring in nursing. They are familiar with Delphi and analytic hierarchy process. The main tasks of the research team include: searching literature at home and abroad, preparing, distributing and collecting expert consultation questionnaires, analyzing and sorting expert opinions, collecting and collating data, and writing papers.

Literature Search
This study searched Cochrane Library, EMbase, PubMed, China Biomedical Medicine (CBM), VIP, CNKI, Wanfang and other Chinese and English databases with "Outpatient", "Nursing indicators", "Nursing quality and indicators", "quality indicators", and Chinese phrases "outpatient", "nursing", "quality indicators, "quality evaluation", and "evaluation indicators" used as key search terms. We collected relevant indicators of outpatient nursing quality in the literature, and combined with the outpatient nursing work standards in the Third-Grade General Hospital Evaluation Standard Implementation Rules (2011) to sort out 67 outpatient nursing quality indicators. With the three dimensions [5] of the nursing quality standard as theoretical framework, we finally formulated three first-level indicators namely structural standard, process standard and effectiveness standard, 13 second-level indicators and 51 third-level indicators.

Preparation of Consultation Questionnaires for Experts
Based on a large number of domestic and foreign literature and interviews with nursing management experts, an expert consultation questionnaire for indicators of outpatient nursing quality was made. The questionnaire mainly included five parts: (1) instructions for filling in the questionnaire, including research purpose, significance and working principle of Delphi method; (2) basic information of experts, including name, age, highest educational background, duty, professional title, and length of employment and other general information; (3) the body part of the questionnaire, including the name of the indicators, the importance of the item (Likert 5-level scoring method), the column for opinions on modifying the indicators, etc.; (4) experts' familiarity with the evaluation of the item and their basis for judgment [6]; (5) table for expert opinion and suggestion.

Inclusion Criteria for Expert
The key to the Delphi method is the selection of experts [7]. Improper selection of experts may increase the deviation of research results [8]. The suitable number of experts is generally 10-50 [9]. Inclusion criteria for this study: (1) experts in the fields of clinical nursing, clinical medicine, nursing management, hospital management and other fields; (2) experts having bachelor degree or above, 15-year or above working experience, and deputy senior or above titles; (3) experts willing to participate in this study, and able to complete 2 rounds of expert consultation.

Implementation of Expert Consultation
The consultation questionnaires were distributed through e-mail or WeChat. The first round was on March 18, 2020, and the questionnaires would be collected within 1 week. To avoid recall bias, the interval between the two rounds of consultation was 20 days [10]. Based on the opinions of the experts at the first round and the statistical results of the data, the research team added, deleted, and modified the contents of the evaluation indicators to prepare the questionnaires to which a summary of the experts' opinions on revision was appended for the second round. The second round of consultation was on April 8, 2020, and the questionnaires would be collected within 1 week. In this study, all indicators meet the selection principle that mean value of importance was not lower than 4.0 (≥4.0), coefficient of variation was not higher than 0.25 (≤0.25), and full score ratio was higher than 22% (>22%) [11]. If this principle was not met, the items would be revised or deleted based on expert opinions and the results of the research team discussion.

Deciding Weight for Indicators of Different Levels
Through Analytic Hierarchy Process Analytic hierarchy process (AHP) is a decision analysis that combines quantitative and qualitative features [12]. The analytic hierarchy process can quantify the basis of subjective judgment so as to improve the effectiveness of decision making. It is suitable for quantitative analysis and determination of indicator weight [13]. The basic principle is to quantify the subjective judgment results of experts, that is, to determine the Saaty scale by the difference of the average value of the importance of the outpatient care quality indicators in the second round of expert consultation, create a judgment matrix, and make consistency test, and finally determine the weight of each indicator and combination weight of outpatient care quality indicators [14]. When the consistency test showed that the CR value was less than 0.10, it indicated that the weight distribution of each indicator was reasonable and had satisfactory consistency [15].

Statistical Methods
We used Excel10.0 to establish a database and statistical software SPSS19.0 for data processing and analysis of Delphi expert consultation results. Rate, mean and composition ratio were used to describe the selection of indicators. The expert positive coefficient was expressed by questionnaire collection rate, authority degree was expressed by authority coefficient, concentration degree was by the mean value of importance of indicators and full score ratio, and variance was by variation and coordination coefficients. AHP was used to quantitatively scale the subjective judgment results of experts, and the statistical software yaahp 10.0 was used to calculate the weight, combination weights and consistency coefficient of indicators.

Basic Situation of Expert Consultation
This study conducted a total of 2 rounds of expert consultation and invited 21 experts from 16 third-grade class A general hospitals in Guangdong, Guangxi, and Jiangxi provinces to participate in the questionnaire survey. There were 15, 4 and 2 experts in nursing management, clinical nursing and hospital management respectively. The age of the experts was (48.81 ± 5.60) and their working years were (28.81 ± 6.42). All experts had more than 15 years of working years. The number of deputy senior titles was 11 (52.38%) and the number of senior titles was 10 (47.62%), of which 5 experts were supervisors for master students, 11 had undergraduate degrees (52.38%), 8 had master degrees (38.10%), 2 had doctoral degrees (9.52%). The data were shown in Table 1.

Positivity, Authority and Coordination of the Experts
A high level of enthusiasm was indicated by ≥70% of expert positivity [16]. The collection rate of consultation questionnaire of the first round was 95.45% (21/22), and the collection rate of the second round was 100% (21/21). The positive coefficients of expert consultation of the two rounds were 0.95 and 1.00 respectively, which indicated high positivity of the experts. The greater the average value of the indicator importance and the full score ratio, the higher the degree of the expert opinion concentration [17] which was shown in the Table 2 and Table 3. The expert authority (Cr) was determined by two factors: the basis for experts to judge the content of the consultation (Ca) and the expert's familiarity with the content of the consultation (Cs) (Cr = (Ca + Cr) / 2, Cr>0.7 means good authority, and Cr>0.8 shows greater familiarity) [10]. In the first round, Ca was 0.895, Cs was 0.867, and Cr was 0.881, and in the second round, Ca was 0.879, Cs was 0.876, and Cr was 0.877, indicating that the experts have a high degree of authority. The degree of coordination of expert opinions refers to whether there is a big difference in the average opinions given by all experts on all indicators. The coordination degree was mainly expressed by the coefficient of variation (CV) and the Kendall coordination coefficient W (Kendall' SW), and the value range of coordination degree was generally from 0.3 to 0.5 [18]. The larger the value of W, the better the degree of coordination of expert opinions. The degree of coordination of expert opinions was shown in Table 4.

Results of Analytic Hierarchy Process
The key step of the combination of Delphi and AHP was to perform the conversion between the Saaty scale and importance value. In this study, the judgment matrix was constructed to determine the Saaty scale by comparing the average importance value of every two indicators. It was assumed that the mean value of the importance of any two indicators in a certain dimension was Zij and Zik. It was stipulated that if the difference between Zij and Zik was 0.25~0.5, Zij was slightly more important than Zik and Saaty scale was 3; if the difference between Zij and Zik was 0.75~1.0, Zij was more important than Zik and Saaty scale was 5; if the difference between Zij and Zik was 1.25~1.5, Zij was strongly more important than Zik and the Saaty scale was 7; if the difference between Zij and Zik was higher than 1.75 (>1.75), Zij was extremely important than Zik and the Saaty scale was 9; if the difference was between the two scales, the Saaty scale was 2, 4, 6, 8. According to this principle, the judgment matrix was constructed, and we strictly followed the steps of analytic hierarchy process. The consistency test of indicators at all levels in this study showed "<0.1" [15], indicating higher consistency in the experts. The judgment matrix of first-level indicators in this study is shown in Table 5.

Indicator Revision Results
According to the results of two rounds of expert consultation, the revision of the indicators is as follows: (1) deleting two third-level indicators: routine outpatient care, effective teamwork; (2) adding one second-level indicator: appointment management; adding seven third-level indicators: emergency response ability of nursing staff, visiting rate of online appointment, rate of breaking online appointment, on-time visiting rate of appointment, one-stop service, communication with and care for patients' family members, and disinfection and isolation facilities; (3) amendment of 8 second-level indicators: modification of nurse manpower deployment to nursing staff deployment, professional ability of nursing staff to comprehensive ability of nursing staff, convenient and quick process to visit process management, implementation of rules and regulations to execution of rules and regulations, etc.; amendment of 16 third-level indicators: required modification of outpatient nurse responsibilities to outpatient nurse qualifications and job responsibilities, nursing manpower deployment to the nurse-patient ratio configuration, and the theoretical knowledge of nursing staff to professional theoretical knowledge of nursing staff. The final evaluation indicators of outpatient nursing quality were determined, including 3 first-level indicators, 14 second-level indicators, and 56 third-level indicators, as shown in Table 6.

Reliable Evaluation Indicators of Outpatient Nursing Quality
Experts' qualifications, positivity, authority and coordination determine the reliability of the results of this study [19]. The 21 experts included in this study have rich clinical practice and theoretical experience, and their professions cover nursing management, clinical nursing, hospital management and many other fields. The experts consulted have worked for more than 15 years (>15), and experts with master's or doctorate degree account for 47.62%, experts with senior professional titles account for 47.62%. The collection rates of the two rounds of expert questionnaires are over 90% (>90%), indicating that the experts are highly active. The authority coefficients of experts of the two rounds are over 0.8 (>0.8), indicating that the experts' opinions are authoritative and the research results are reliable. The coefficients of variation of all indicators in the results of the two rounds of expert consultation are less than 0.25 (<0.25), and the Kendall coordination coefficients are 0.303 and 0.313 which are between 0.3~0.5 (P <0.001), which suggests that the opinions of experts tend to be consistent, indicating high recognition and coordination.

Scientific Evaluation Indicators of Outpatient Nursing Quality
This study extracts quality indicators of outpatient care based on a large number of domestic and foreign literature and interviews with experts, and uses the three dimensions of nursing quality standards (structural standards, process standards, and effectiveness standards) as the theoretical framework to establish the logical relationship between each indicator. Representative and authoritative experts are selected according to the expert inclusion criteria, and the steps of the Delphi method is strictly followed for expert consultation. Expert opinions of the two rounds are collected and summarized, and 3 first-level indicators, 14 second-level indicators and 56 third-level indicators finally come out. Combination of the Delphi method and the analytic hierarchy process combines quantitative and qualitative methods, avoiding subjective deviations caused by the only use of Delphi expert consultation, making the research results more scientific and reliable [6].

Structural Standard
The structural standard refers to the basic structure of the medical service organization [20]. It includes three second-level indicators namely nursing staff deployment, outpatient system norms and outpatient organization structure, and 16 third-level indicators. Among the three second-level indicators, outpatient system norms have the largest weight, 0.4934, which indicates that the rules and regulations are an important component of nursing management and hospital management. The outpatient clinic is a window of the hospital to face the society. There are many patients and complicated diseases. Sound outpatient system norms are a prerequisite for preventing nursing errors and accidents, and also a powerful guarantee for preventing nurse-patient disputes, which is consistent with the research results of Li Yumei, et al. [21]. The results are similar. If there is no effective system and standard for outpatient management, the management level of the hospital will be greatly reduced. The three-level indicators in the norms of the outpatient system are mainly formulated around the nursing service norms and work standards, and the outpatient process guidelines account for the largest weight (0.3249), which shows that with the progress of society and increasingly higher requirements of patients for medical experience, medical institutions are also required to continuously develop standardized and thorough outpatient procedures guidelines to save time, and provide services of better quality and more convenience for patients so as to improve the efficiency of medical treatment and satisfaction of patients. This is similar to the research results of Han Huifang et al. [22]. Among the second-level indicators under the structural standard frame, nursing staff deployment has the second largest weight value, 0.3108, ranking behind the outpatient system norms. It shows that the deployment of nursing staff is the prerequisite for ensuring the quality of outpatient care. The proportion of nurses and patients in China has been imbalanced for a long time. Research shows that nearly 31% of hospitals do not meet the requirement that more than one nurse responsible for 100 patients [23], which also suggests that nursing managers should optimize human resource management and guarantee the quality of nursing.

Process Standard
The process standard refers to the process during which medical personnel provide service in accordance with work norms [20]. The weight value of the process standard in this study is 0.297, which has most indicators among the three first-level indicator systems, mainly including 6 second-level indicators and 27 third-level indicators. It shows that the process standard is a key part in the process of monitoring the quality of outpatient nursing work and has prospective control of quality, playing an important role in ensuring the quality of outpatient nursing. The second-level indicator that has largest weight in the process standard section is the comprehensive ability of nursing staff (0.2909). Experts believe that outpatient nursing work involves a wide range of aspects, and thus the nursing staff need not only good basic quality, but also strong professional and service ability to ensure the quality of care, which is consistent with the results of Ai Rong, et al. [2]. This also suggests that nursing managers should pay attention to the continuing education of nurses to improve the comprehensive quality of outpatient nurses [24]. Among the third-level indicators under the comprehensive ability of nursing staff, the top three weights are emergency response capacity of nursing staff (0.3614), nursing staff's communication skills (0.2526) and nursing staff's service attitude (0.1765). Because outpatient department are confronted with various complicated diseases and unpredictable factors, nurses are required to have stronger emergency response capabilities in case of emergency [25] so as to ensure that in the process of visits, effective preventive measures and treatment for possible accidents are taken to ensure the quality of medical treatment and safety of patients. The outpatient clinic is the first place for patients to receive medical treatment where there may be inevitable anxiety and irritability because patients and their families are eager to seek medical treatment. If nursing staff are proficient in communication skills, they can effectively avoid language conflicts with patients or their families and thus reduce the occurrence of medical disputes [26].

Effectiveness Standard
The effectiveness standard refers to the effect presented by the medical staff after providing various interventions for the patients [20]. The weight of the effectiveness standard in this study is 0.5396, including 5 second-level indicators and 13 third-level indicators. The top two second-level indicators in terms of weight value in the effectiveness standard section are the outpatient hospital infection management (0.3614) and patient's satisfaction and medical experience (0.2526). The control management of outpatient hospital infection is very important, which is related to nursing safety. The third-level indicator of highest weight in the section of outpatient hospital infection management is hand hygiene compliance (0.3905) which receives high degree of expert recognition, indicating that outpatient nurses' hand hygiene compliance is an important part of effective control and management of outpatient hospital infection. This also indicates that nursing managers should continue to strengthen training on knowledge of hospital infection and improve the hand hygiene compliance of nursing staff so as to ensure the safety of patients. The third-level indicator with the highest weight in the section of patient's satisfaction and medical experience is patient's satisfaction (0.5396), followed by patient's waiting time (0.297). It shows that patient's satisfaction is the core content of the effectiveness standard, an important indicator that reflects the quality of outpatient care and patient's medical experience [26].

Conclusions
A unified and standardized nursing quality indicator system is highly comparable, which facilitates communication between different hospitals, and also provides a basis for the development of relevant nursing quality improvement strategies at the national level [27]. A scientific quality assessment tool is the prerequisite for ensuring the quality of nursing. Outpatient nursing quality indicators, as a quantitative measurement tool, play an important role in the construction of nursing quality system. The quality of outpatient care is closely associated with patient's satisfaction, medical experience and social benefits of the hospital. It is necessary to manage the quality indicators of the outpatient service and continuously improve the quality of care. This study uses the Delphi method and the analytic hierarchy process to construct outpatient nursing quality evaluation indicators, which can provide standard quantitative measurement tools for nursing managers in the construction and evaluation of outpatient nursing quality. The indicators can also guide outpatient nursing staff to perform nursing service according to the quality indicators and at the same time help nursing managers to evaluate and continuously improve the quality of outpatient nursing services according to the requirements of quality evaluation indicators, promoting the scientific and standardized management of outpatient nursing quality.