Determining Cutting Points of the Maslach Burnout Inventory for Nurses to Measure Their Level of Burnout Online
Huan-Fang Lee1, 2, 3, Hui-Ting Kuo4, Cheng-Li Chang4, Chia-Chen Hsu4, Tsair-Wei Chien5, 6, *
1Departent of Nursing, National Cheng Kung University Hospital, Tainan, Taiwan
2Nursing Department, Chung Hwa University of Medical Technology, Tainan, Taiwan
3Nursing Department, National Cheng Kung University, Tainan, Taiwan
4Departent of Nursing, Chi-Mei Medical Center, Tainan, Taiwan
5Department of Medical Research, Chi-Mei Medical Center, Tainan, Taiwan
6Department of Hospital and Health Care Administration, Chia-Nan University of Pharmacy and Science, Tainan, Taiwan
To cite this article:
Huan-Fang Lee, Hui-Ting Kuo, Cheng-Li Chang, Chia-Chen Hsu, Tsair-Wei Chien. Determining Cutting Points of the Maslach Burnout Inventory for Nurses to Measure Their Level of Burnout Online. History Research. Vol. 5, No. 1, 2017, pp. 1-8. doi: 10.11648/j.history.20170501.11
Received: December 19, 2016; Accepted: January 21, 2017; Published: February 24, 2017
Abstract: This study is to determine cutting points for the Chinese version of the MBI-HSS and to design an online assessment tool that instantly measures a nurse’s burnout level. We illustrate (1) the traditional way for determining the cutting points of a scale when the binary classification groups was still known, and (2) the norm-reference approach without groups of binary classifications was used to determine the cutting points on three subscales for the MBIO-HSS. An online MBIO-HSS assessment APP for smartphones was incorporated with the cutting points to instantly display the level of burnout for nurses. The cutoff points of the MBI-HSS were ≤ 21 and ≤ 32 for the Emotional subscale, ≤ 23 and ≤ 30 for the Reduced Personal Accomplishment subscale, ≤ 6 and ≤ 12 for the Depersonalization subscale, and ≤ 15 and ≤ 17 (i.e., low, moderate, and high level) for the overall scores. An available-for-download online MBI-HSS APP for nurses was developed and demonstrated.
Keywords: Nurse Burnout, MBI-HSS Chinese Version, Cutting Points, Prevalence
Burnout is a critical syndrome and problem in high-tech service-oriented societies, especially for nurses in healthcare settings [1-3]. Many studies have addressed that burnout affects employee’s physical and psychological status [4-6], institute well-being [7-10], and indirect to the patient outcomes if the professionals are nurses [5, 9]. According to Maslach , burnout is a syndrome of emotional exhaustion (EE), reduced personal accomplishment (PA), and depersonalization (DP) that can occur in individuals who work much more with people-related jobs such as healthcare and education.
1.1. The Maslach Burnout Inventory
The Maslach Burnout Inventory-Human Services Survey (MBI-HSS)  is an instrument that has been used most widely for measuring the burnout climate of healthcare providers [10, 13-15]. The original version of the MBI-HSS is a 22-item domains with a 7-point scale (from never = 0 to every day = 6) to measure burnout for workers in a recent week. The three subscales of burnout include nine items for EE, eight opposite items for PA, and five items for DP. Despite its popularity in social science, not only does the factor structure (i.e., the item-total correlation based on their variances) differ between cultures and healthcare among provider systems [14, 16-20], but so do the cutting points substantially differ between each other. Maslach et al. , accordingly, claimed that the levels of burnout (low, moderate, high) with their respective cutting points might differ across countries. Schaufeli and Van Dierendonck  suggested that nation-specific and clinically derived cutting points that should be determined and used for healthcare providers so as to easily comoare each other.
Maslach and Jackson  said that the subscale total scores were 54 for EE, 48 for PA, and 30 for DP. They also claimed that the level of burnout was high if EE was ≥ 27, PA was ≤ 21, and DP was ≥ 13; moderate if EE was 17-26, PA was 38-22, and DP was 7-12; and low if EE was ≤ 16, PA was ≥ 39, and DP was ≤ 6. Schaufeli and Van Dierendonck  wondered whether these three sets of scores were arbitrary when the three groups contained an equal number of members . Maslach and Jackson  suggested that valid criteria are in urgent need required to classify levels of burnout for use in literature, but no any one till now is present in published papers. This is, no large and rigorous quantitative studies have reported cutting points that can objectively classify levels of burnout and can be generalized to other worksites and other samples within a relatively homogeneous nation.
1.2. Cutting Points of the MBI-HSS Are Required
Many papers report burnout scores for individuals but translating the numerical scores into the degree or type of their burnout relative to cutting points. The purpose is to be possible for other worksites and other ethnic samples in comparison. However, the burnout golden standard (i.e., ensuring the binary classification groups before the study) is costly, is subject to a small sample size, and would, therefore, be inappropriate generalize to any other population of interest. If we use population information generalized by study sample analyzed to determine cutting points, the result can theoretically be more widely used for other worksites and other ethnic samples than can traditional approaches directly analyze ethnically or culturally specific study samples.
1.3. Online Assessment Using Smartphones Is Required
As with all forms of Web-based technology, advances in mobile health (mHealth) and health communication technology are rapidly increasing . Till now, there is no any online APP for smartphones that measures nurse burnout levels in healthcare industry. If the cutting points are used with the MBI-HSS, the online assessment can thus alert individual APP users to alleviate their mental strain before it becomes a serious burnout problem.
1.4. Study Aims
The aims of the current study were thus to (i) determine the cutting points of the MBI-HSS person strata according the literature, and (ii) design an online burnout assessment APP for smartphones.
2.1. Data Source
The data was collected from three different levels hospitals in Southern Taiwan. Staff nurses (n = 1,000) were recruited, and 970 (93%) completed the MBI-HSS Chinese version.
The current study was approved and monitored by the Chi Mei Medical Center Institutional Review Board before we began to retrieve data. All hospital and study participant identifiers were stripped from the data.
The factor structure of the MBI-HSS for nurses in Taiwan was examined using exploratory factor analysis and confirmatory factor analysis . The modified factor structure included the original three factors (i.e., subscales) with 20 items (i.e., removing items # 14 and #22  from the MBI-HSS). The three subscales of burnout consist of 8 items for EE, 8 opposite items for PA, and 4 items for DP. The subscale total scores for EE, PA, and DP were 48, 48, and 24, respectively. The indices of the model fit were GFI = 0.92, AGFI = 0.90, and RMSEA = 0.05. The data collected using the MBI-HSS Chinese version  were used to determine the cutting points in the current study.
2.4. Traditional Method for Determining Cutting Points
Traditionally, researchers in clinical practice use ROC (receiver operating characteristic) curves to plot the true-positive rate (sensitivity) against the false-positive rate (1 - specificity) at various threshold settings  (Figure 1). The preliminary condition is to know the patient’s classification (i.e., stratum) (e.g., separating person burnout strata with low, moderate, and high level) before conducting the ROC.
However, we usually do not know the patient’s true- and false-positive disease-specific status, as in this study, unless we have done another costly diagnostic intervention to obtain the so-called gold standard test (e.g., a cutting point) before doing a study. The area under the ROC curve is computed by the formula: . The cutting points are determined at the maximum of the summation of sensitivity and specificity (Table 1 last column).
Note. a: Area under ROC curve (AUC) summing the area of trapezoid with curved edge referred to columns F and G; b: Sensitivity+ Specificity; c: cutting point; c:summing values of the total rows: d: the maximum of the summation of sensitivity and specificity.
2.5. A Norm-Reference Approach for Determining Cutting Points
According to the literature [26-28], as a scale’s reliability (i.e., Cronbach’s a) increases, so does the person-number of ranges that can be confidently distinguished. Measures from two instruments with reliabilities of 0.67 will tend to vary within two groups that can be separated with 95% confidence; 0.80 will vary within three groups; 0.90, within four groups; 0.94, within five groups; 0.96, within six groups; 0.97, within seven groups; and so on .
To compute the number of the strata, pick up any two adjacent normally distributed samples using the Microsoft Excel function = NORMDIST (mean, standard deviation [SD], TRUE); the mean is the cluster center obtained using the k-mean method when the number of strata is known according to the Cronbach’s a scale , and the SD is obtained from the individual scores of the specific cluster. Using a brute force search of the two adjacent samples, the cutting points can be determined at the maximal summation of specificity and sensitivity across all possible scores.
2.6. An Online Burnout Assessment APP Was Designed for Use on Smartphones
An online routine was designed for patients to report their burnout scores. In addition to 20 items from a previous study , we added three items from a new scale of Satisfaction with Nursing (SN) . High scores indicate more satisfaction with being a nurse. All the opposite responses are automatically transferred to a burnout score (e.g., the higher the score on the PA and the SN scales, the greater the tendency for burnout using the formula of 6 – response). The overall score is the mean of the EE, PA, and DP scale scores. There are five bins to record the count of interest for the study bins (i.e., EE, PA, DP, SN, and overall).
To help examinees recognize whether burnout is present, we set two indices for ensuring the occurrence: (i) the revised Ferguson (δ) coefficient [31-35] (= in a range from 0 to 1, and (ii) the c2 (from zero to infinite, where g denotes the five bins, the observed , , the Expected represents the first cutting point (i.e., the upper limit of low-level burnout) of each bin; scorei, and pscore are the response summation score and the possible total summation score of each bin, respectively. When the δ > 0.9 (toward a uniform distribution [31, 35]) and c2 < 3.84 (= 1.962) are concurrently true, we confirm that the examinee is at the low burnout level. Otherwise, the responses might be distraction, careless, mistaken, cheating, or awkward if the δ < 0.9 or less. That is because the score ratios across subscale are not inconsistent (i.e., toward a non-uniform distribution [[31, 35]]).
2.7. Statistical Tools and Data Analyses
SPSS 15.0 for Windows (SPSS Inc., Chicago, IL) was used to calculate Cronbach’s a on the three subscales of response datasets. The comparisons between demographic variables were measured using descriptive statistics, c2, and analysis of variance (ANOVA). Data were analyzed using SPSS and Microsoft Excel. Cutting points were determined at maximal summations of specificity and sensitivity for each person stratum when strata central points were determined using k-mean cluster analysis.
The sample of 970 nurses was obtained from the study. The mean age of the participants was 31 ± 4.3 years, more than 99% (n = 962) were female, more than 74% (n = 725) were unmarried (Table 2).
3.1. Cutting Points of MBI-HSS Chinese Version
The Cronbach’s a of the three-burnout subscales ranged from 0.74 to 0.84 in the current study. Thus, the number of person strata for the MBI-HSS Chinese version can be divided into three groups: low, moderate, and high. For each subscale, ANOVA showed significant differences (p < 0.001) between the three levels. The cutting point sensitivity and specificity ranged from 0.82 to 0.97. The area under the curve (AUC) ranged from 0.97 to 0.99 (Table 3).
|Subscale||Cronbach’s α||Cluster centers||ANOVA F-values||Cutting point§||Sensitivity||Specificity||AUC|
|Emotional exhaustion||0.84||17, 27, 38||3810*||21||0.94||0.94||0.99|
|Depersonalization||0.83||4, 10, 16||3742*||6||0.95||0.95||0.99|
|Reduced Personal accomplishment||0.74||11, 21, 28||3168*||23||0.94||0.94||0.99|
ANOVA: analysis of variance; AUC: area under the curve; § the scores for personal accomplishment have been reversed to the burnout tendency; *p < 0.05.
3.2. Online Burnout Assessment
By scanning a QR-code (Figure 2, top right, bottom) or downloading the APP, the burnout questionnaire appears on the smartphone. We developed a mobile survey procedure to provide a practical demonstration of the newly designed burnout MBI-HSS application in action. The burnout APP processed each nurse item-by-item with audio and visual (Figure 2, top left). The result with a high burnout level across all subscales instantly shows on a smartphone (Figure 2, bottom). The δ index is 1.0 (> 0.9), which means that the scores of the five bins are equal ratios to their respectively corresponding cut-point criteria, and c2 is 21.77 (> 3.84), which means that the scores of the five bins are different from the low burnout level. These index values indicate that the examinee is not at the low burnout level.
4.1. Key Findings
The cutoff points of the MBI-HSS Chinese version were ≤ 21 and ≤ 32 for Emotional, ≤ 23 and ≤ 30 for Negative Personal Accomplishment, ≤ 6 and ≤ 12 for Depersonalization, and ≤ 17 and ≤ 15 (i.e., low, moderate, and high level) for the overall scores. An available-for-download online MBI-HSS APP for nurses was suited for smartphones.
4.2. Additional Contribution to Existing Research
The MBI-HSS has been used most widely for measuring burnout in the world [10, 13-15]. The psychometric properties of 20 items of these scales have been validated for use in hospital nurses . However, most of them merely report numerical results that are not translated into the degree (or classification) of their burnout problems relative to a cutting point that can be generalized to other worksites and other ethnic samples.
Maslach and Jackson  reported the level of burnout using the criteria of cutting points. Schaufeli and Van Dierendonck  disagreed with their methods and wondered why these three cutting points were arbitrary and merely based on to an equal sample size of the three groups divided by the high, moderate, and low levels of burnout .
Schaufeli & Janczur  reported that staff with similar characteristics in Europe undergo lower rates of exhaustion and depersonalization than do staff in North America, and suggested different cultural values as a possible explanation for the different rates. However, all of which should translate those numerical results into the degree (or classification) of their burnout problems. Furthermore, Maslach et al.  confirmed that levels of burnout must be different in various countries. However, we have not found any research that reported the cutting points used for the MBI-HSS on hospital nurses and suitable for a nation-based reference when the binary classification groups of burnout were unknown, and no application that incorporates the MBI-HSS has been used for smartphones.
4.3. What It Implies and What Should Be Changed
We have provided a way to determine the cutting points of person burnout strata using a norm-referred method. It is because we usually do not know the nurse’s true- and false-positive status. Thus, many studies in their limitations sections caution that their results cannot be generalized to other sites or to other types of sample groups. This is because the data were sample-dependent. How we estimate the population properties using the sample data (e.g., Cronbach’s a coefficient for a scale ) before determining cutting points and then make inferences (e.g., the cutting points for a scale) about the population is the main feature of the present study.
Thus, in this study, the norm-referred method was introduced based on suggestions in the literature [26-28]. The cutting points were theoretically determined using an inference based on the study sample. Future studies are suggested to use this way to determine cutting points for other diseases in healthcare settings.
4.4. Strengths of This Study
It is easy to set up the online burnout assessment form if the designer uploads relevant audio and visual files to the corresponding questions of the database. We especially developed two indices (i.e., the revised Ferguson Delta and the c2) for helping users (or psychiatrists) discriminate the level of burnout according the graphical result in Figure 2: the higher Delta is, the more confident we are that the responses are not careless, mistaken, cheating, or awkward; the higher the value of c2 (> 3.84), the more likely is moderate or even high burnout (e.g., not at the low burnout level).
As with all forms of Web-based technology, advances in mobile health (mHealth) and health communication technology are rapidly emerging . Mobile online burnout assessment is promising and worth considering in many fields of health assessment.
On the other hand, health literacy is increasingly recognized as critical . Adults with below basic or basic health literacy are more likely than adults with higher health literacy to get information about health issues from multimedia graphical representations [39,40] rather than traditional newspapers, magazines, books, brochures, or pamphlets. An online burnout assessment such as the one we developed can be used to inform examinees quickly about when and whether they should take actions or follow-up to see a psychiatrist, and how to improve their behaviors and attitudes given their lifestyle is not changed. Mobile online burnout assessment is promising, and it is worth using for promoting nurses’ health literacy. Interested readers are recommended to scan the QR-code or download the APP in Figure 2 to practice it in their own way and to conduct an online burnout assessment.
4.5. Limitations and Future Studies
Our study has some limitations. First, although we believe that the online burnout assessment is suitable for any normally distributed group, there is no evidence to support our assumption that our sample of nurses is fully normally distributed, which might affect what cutting points are determined. This means that more than one stratum is required if data are not normally distributed . We recommend additional studies using samples with a variety of distributions to see whether different cutting points are arrived.
Second, although the MBI-HSS Chinese version’s Cronbach’s a coefficients ranged from 0.74 to 0.84 , we conservatively determined that all the subscales’ nurse strata were three instead of four or more when Cronbach’s a for a scale approached 0.90 . It is convenient, clear, and simple to show the stack bar chart plot, like the one in Figure 2, of a graphical representation for users.
Third, the study was based on a previously published paper . All of the data were sampled from that study. If any steps or nodes were incorrect, the result would be different from the real situation.
Forth, the MBI-HSS is a three-dimensional construct. The CAT should be applied to the so-called multidimensional CAT (i.e., MCAT) instead of using the sequential CAT(i.e., one CAT following another CAT) procedure like this study. Future studies are recommended to apply MCAT to develop the online MBI-HSS CAT.
Finally, the study sample was taken from Taiwanese data in a nurse survey. The cutting points for the MBI-HSS Chinese version are just suited for Chinese (particularly for Taiwanese) society in healthcare settings. Generalizing these burnout assessment findings might be somewhat limited because the sample consisted only of nurses working for inpatients. Additional studies are needed to reexamine whether the psychometric properties of the burnout assessment are suitable for other worksites in a hospital.
We found that the online mobile module used for smartphones is promising for assessing nurse burnout. The visualized presentation of the burnout assessment shows that it is innovative and novel for helping hospital nurses (or psychiatrists) measure the level of burnout online in clinical settings.
The authors declare that they have no competing interests.