Quality Evaluation Method of Dissertation for Master Degree Based on AHP
Lianhong Ding1, *, Zhaozhe Cui2
1School of Information, Beijing Wuzi University, Beijing, China
2School of Logistics Engineering, School of Information, Beijing Wuzi University, Beijing, China
To cite this article:
Lianhong Ding, Zhaozhe Cui. Quality Evaluation Method of Dissertation for Master Degree Based on AHP. Education Journal. Vol. 5, No. 6, 2016, pp. 142-146. doi: 10.11648/j.edu.20160506.13
Received: October 15, 2016; Accepted: October 26, 2016; Published: November 15, 2016
Abstract: How to evaluate the quality of a dissertation for master degree is important for the postgraduate education. The evaluation of dissertation is a complex assignment. The evaluation process in certain extent is accompanied by randomness, fuzziness, uncertainty/intangible and instability. The Analytic Hierarchy Process (AHP) is a multiple criteria decision-aiding method developed by Saaty. In this paper a quantitative method for the evaluation of dissertation is put forward based on the analytic hierarchy process. Multiple evaluation criteria and sub-criteria are organized into a hierarchy. The relative importance of these criteria is accessed and different weight values are assigned to evaluation criteria. As the result, the evaluation process of the dissertation for master degree becomes more simple and convenient. The evaluation result is more objective and exact.
Keywords: Quality Evaluation, Dissertation, AHP, Master Degree, Higher Education
Master thesis can reflect the status of postgraduate education. The quality of a dissertation for master degree directly determines whether a learner can receive the master degree. It is also an important criterion to measure the level of postgraduate education. Building an appropriate quality evaluation system for dissertation for master degree has become an important assignment for the postgraduate education. Good evaluation system forms the basis not only for a marker to make various judgments in allocating a degree to the learner, but also suggests what the learner is required to do for a degree.
Quality improvement in higher education has been addressed in different ways in the literature and different articles. The assessment elements may include assessment means, content, criteria and weighting. Clear description of assessment items and criteria can encourage the learner actively engage with meeting the criteria. It is also important to understand the mechanisms between evaluation elements and final mark allocated to the thesis .
Individual thesis contents are different from each other when the research project are completed. One thesis may focus on one aspect while others on other aspects. As a result, a thesis is unique in terms of topic, research/knowledge area, or/and analysis skill required. When theses are examined, some inconsistency arises from various sources such as unclear guidelines for assessment items and criteria, examiner expertise, etc. The inconsistency sometime causes serious impacts on students in various ways. The existing evaluation process in certain extent is accompanied by randomness, fuzziness, uncertainty/ intangible and instability.
The Analytic Hierarchy Process (AHP) is a multi-criteria theory of measurement. It addresses the issue of how to structure a complex decision problem, identify its criteria, measure the interaction among them and finally synthesiseall the information to arrive at priorities, which depict preferences . The AHP is able to assist us to obtaining a detailed understanding of the evaluation criteria for the dissertation.
This paper put forward a quantitative method for the evaluation of dissertation for master degree. In order to avoid the randomness, fuzziness, uncertainty/intangible and instability, the AHP is adopted. Multiple evaluation criteria and sub-criteria are organized into a hierarchy. The relative importance of these criteria is accessed and different weight values are assigned to evaluation criteria. As the result, the evaluation process of the dissertation for master degree becomes more simple and convenient. The evaluation result is more objective and exact.
2. Related Work
A handbook for linking levels, learning outcomes and assessment, believes that the use of effective assessment criteria is a long way behind the learning outcomes and level descriptors . Generally the thesis evaluation criterion may be broken into two quantity and quality, which can be applicable to any assessment. No matter which kind of evaluation system is adopted, a set of evaluation criteria should be selected first.
If the evaluation method is qualitative, each review expert is asked to give a short description for each criterion and a grade for the whole thesis such as good, acceptable or unacceptable. Lovitts gives guidelines for reviewer how to allocate a final grade for a dissertation based on qualitative assessment system. The grade may be outstanding, very good, acceptable and unacceptable . It is stated that many dissertation-committee members can recognize a quality dissertation when they see it, adding that they can describe general characteristics of outstanding, very good, acceptable and unacceptable dissertations. Webster et al discussed some common general criteria and their ambiguity for the dissertation assessment .
Arefi et al discusses the quality of master’s degree program from the requirements of students. They classifies the quality requirements into four categories of attractive, one-dimensional, must-be and indifferent. Finally, better value and worse value were measured. But none deals with quantitative nature of criteria . Indeed, no fundamental framework for assessment linking assessment items and quantitative criteria has been found in related literature. Woolf points out that there are no explicit evidences indicating whether some criteria are more important than others or markers may weight the criteria .
If the evaluation system is quantitative, each criterion is specified by a weight value. Review expert gives a mark for each criterion. The final score for the thesis is calculated according to the weight value specified to the criteria and mark given by reviewer. The evaluation system of thesis should be explicit and consistent. Brits et al focus on the evaluation of the extensive mini-dissertations in the master’s degree in family medicine. Different weight values are specified to 12 assessment criteria and the coefficient of variation was used to compare variations between the different criteria . Freeman and Lewis introduced principles of good assessment criteria mainly in terms of presentation for the dissertation . A hierarchical structure of assessment elements is proposed for thesis assessment weight calculation. Selection principles of thesis assessment components and criteria were discussed as well . Khambadkone select different assessment criteria for different stages of assessment. Different weight values are specified to different stage of assessment as well .
The AHP is one of the most commonly used methods of assessment. The AHP handles complex decision-making problem by structuring the complex problem into a simple and comprehensible hierarchical structure. Ji et al, implement the assessment of network security through AHP. The security policies are weighted by a hierarchical structure . Lirn et al adopt AHP to weight the criteria for port selection. These criteria include port efficiency, adequate infrastructure, frequency of ship visits, and so on . AHP also is used to weight the assessment criteria for the project management by Al-Harbi .
3. Evaluation System of Dissertation for Master Degree
In China, there are two prevalent kinds of evaluation methods for quality of master thesis. One is qualitative. The other is quantitative. No matter which kind of evaluation system is adopted, a set of evaluation criteria should be selected first. Then if the evaluation method is qualitative, each reviewer is asked to give a short description for each criterion and a grade for the whole thesis such as outstanding, good, acceptable and unacceptable. If the evaluation system is quantitative, each criterion is specified by a weighting value. Reviewer gives a mark for each criterion. The score for the whole thesis is calculated according to the weighting and mark of each criteria.
The quantitative method is more objective than qualitative method. In order to build a quantitative evaluation system for the master thesis two essential issues must be resolved. First of all, a set of criteria must be selected for dissertation evaluation. Then weight values are allocated to corresponding criteria. The criterion with bigger weight value contributes more for the description of the quality for the dissertation.
Evaluation Criteria. Usually, criteria are organized into two levels. First the fundamental criteria such as topic selection, thesis writing and contribution are found. Then each criterion of them is refined into sub-criteria in fine grained. For example, the thesis writing can be refined as content, structure, methodology and normalization.
Weight Coefficient Assignment. Determine the weight coefficient is an important step for the building of the quality evaluation system. To a great extent, weight value for each criterion effects the evaluation of the degree thesis. The rationality and accuracy of the weights directly influence the reliability of the evaluation results. The quality evaluation for the master dissertation can be regarded as a kind of multiple decision problem. The analytic hierarchy process (AHP) is a decision support tool for complex multi-criteria decision problems. This paper adopts AHP to assign weighting values to evaluation criteria.
4. Analytic Hierarchy Process (AHP) Decision Model
"Analytical Hierarchy Process (AHP) is an approach to decision making that involves structuring multiple choice criteria into a hierarchy, assessing the relative importance of these criteria, comparing alternatives for each criterion, and determining an overall ranking of the alternatives" . Applicability, simplicity and low labor-intensity are three obvious advantages of AHP. As introduced before, AHP can be applied for multiple criteria decision making and weight coefficient assignment for criteria. The basic steps of AHP for multiple criteria decision making are as following:
(1) Define the problem and determine its goal.
(2) Structure the hierarchy from the top (the objectives from a decision-maker's viewpoint) through the intermediate levels (criteria on which subsequent levels depend) to the lowest level which usually contains the list of alternatives.
|Numerical rating||Verbal judgments of preferences|
|8||Very strongly to extremely|
|7||Very strongly preferred|
|6||Strongly to very strongly|
|4||Moderately to strongly|
|2||Equally to moderately|
(3) Construct a set of pair-wise comparison matrices for each of the lower levels with one matrix for each element in the level immediately above by using the relative scale measurement shown in Table 1. The pair-wise comparisons are done in terms of which element dominates the other.
(4) There are n(n-1)/ judgments required to develop the set of matrices in step (3). Reciprocals are automatically assigned in each pair-wise comparison.
(5) Hierarchical synthesis is now used to weight the eigenvectors by the weights of the criteria and the sum is taken over all weighted eigenvector entries corresponding to those in the next lower level of the hierarchy.
(6) The consistency is determined by using the eigenvalue, λmax, to calculate the consistency index, CI=(λmax-n)/(n-1), where n is the matrix size. Judgment consistency can be checked by taking the consistency ratio (CR) of CI with the appropriate value in Table 2. The CR is acceptable if it is lower than 0.10. Or else the judgment matrix is inconsistent.
|Size of matrix||1||2||3||4||5||6||7||8||9||10|
5. AHP Based Weighing Method for Evaluation Criteria
Here AHP is used for assign weight coefficient for evaluation criteria. Figure 1 is the hierarchy built for criteria weighting. The hierarchy is consisted of overall objective and selection criteria. Four fundamental criteria and twelve sub-criteria are selected to evaluate the quality of dissertation for master degree.
We compared the criteria and sub-criteria according to their relative importance with respect to the parent element in the adjacent upper level. The judgment matrices for criteria and sub-criteria are given in table 2-table 6. The consistency ratios of those are 0.0000, 0.0000, 0.0687, 0.0688 and 0.0000 respectively. The values of the consistency ratios are all within the acceptable range of 0.10.
|Quality of dissertation for master degree||A1||A2||A3||A4|
The weighting result for the criteria and sub-criteria is listed in table 8.
|Criteria||Weight value||Sub-criteria||Weight value|
|A1-Topic selection||0.1244||B11-Theoretic foundation||0.0858|
|A2-Research ability||0.4131||B21-Basic theory and specialized knowledge||0.0502|
|B22-Ability to work in scientific research||0.0613|
|A3-Writing ability||0.2769||B31-Literature review||0.1457|
|B32-Organization of the thesis||0.0700|
|B33-Standardization on writing||0.0612|
6. Case Study and Conclusion
According to the evaluation criteria selected and the weight coefficient assigned by AHP, the score for the whole thesis is calculated according to the weighting and mark of each criteria. As the result, the quality evaluation result of a dissertation for master degree is calculated as: B11×0.0858+B12×0.0386+B21×0.0502+B22×0.0613+B23×0.1508+B24×0.1508+B31×0.1457+B32×0.0700+B33×0.0612+B41×0.0784+B42×0.0642+B43×0.0430.
There are two dissertations, d1 and d2, for master degree. The marks of sub-criteria for d1 are 70, 80, 68, 70, 70, 70, 75, 78, 75, 67, 70 and 68. The marks of sub-criteria for d2 are 85, 78, 83, 78, 82, 87, 75, 81, 78, 77, 70 and 78. According to our method, the quality score for d1 and d2 are 72 (71.5589) and 80 (79.9929) respectively.
From the case study given above, we can find our method makes the assignment of the reviewer simple and clear, the calculating of the quality score for the dissertation objective and fair.
This work was supported by Beijing Key Laboratory (NO: BZ0211), Beijing Intelligent Logistics System Collaborative Innovation Center and Reform Project in BWU Postgraduate Education (054160584407).