Education Journal
Volume 5, Issue 6, November 2016, Pages: 142-146

Quality Evaluation Method of Dissertation for Master Degree Based on AHP

Lianhong Ding1, *, Zhaozhe Cui2

1School of Information, Beijing Wuzi University, Beijing, China

2School of Logistics Engineering, School of Information, Beijing Wuzi University, Beijing, China

Email address:

(Lianhong Ding)

*Corresponding author

To cite this article:

Lianhong Ding, Zhaozhe Cui. Quality Evaluation Method of Dissertation for Master Degree Based on AHP. Education Journal. Vol. 5, No. 6, 2016, pp. 142-146. doi: 10.11648/j.edu.20160506.13

Received: October 15, 2016; Accepted: October 26, 2016; Published: November 15, 2016


Abstract: How to evaluate the quality of a dissertation for master degree is important for the postgraduate education. The evaluation of dissertation is a complex assignment. The evaluation process in certain extent is accompanied by randomness, fuzziness, uncertainty/intangible and instability. The Analytic Hierarchy Process (AHP) is a multiple criteria decision-aiding method developed by Saaty. In this paper a quantitative method for the evaluation of dissertation is put forward based on the analytic hierarchy process. Multiple evaluation criteria and sub-criteria are organized into a hierarchy. The relative importance of these criteria is accessed and different weight values are assigned to evaluation criteria. As the result, the evaluation process of the dissertation for master degree becomes more simple and convenient. The evaluation result is more objective and exact.

Keywords: Quality Evaluation, Dissertation, AHP, Master Degree, Higher Education


1. Introduction

Master thesis can reflect the status of postgraduate education. The quality of a dissertation for master degree directly determines whether a learner can receive the master degree. It is also an important criterion to measure the level of postgraduate education. Building an appropriate quality evaluation system for dissertation for master degree has become an important assignment for the postgraduate education. Good evaluation system forms the basis not only for a marker to make various judgments in allocating a degree to the learner, but also suggests what the learner is required to do for a degree.

Quality improvement in higher education has been addressed in different ways in the literature and different articles. The assessment elements may include assessment means, content, criteria and weighting. Clear description of assessment items and criteria can encourage the learner actively engage with meeting the criteria. It is also important to understand the mechanisms between evaluation elements and final mark allocated to the thesis [1].

Individual thesis contents are different from each other when the research project are completed. One thesis may focus on one aspect while others on other aspects. As a result, a thesis is unique in terms of topic, research/knowledge area, or/and analysis skill required. When theses are examined, some inconsistency arises from various sources such as unclear guidelines for assessment items and criteria, examiner expertise, etc. The inconsistency sometime causes serious impacts on students in various ways. The existing evaluation process in certain extent is accompanied by randomness, fuzziness, uncertainty/ intangible and instability.

The Analytic Hierarchy Process (AHP) is a multi-criteria theory of measurement. It addresses the issue of how to structure a complex decision problem, identify its criteria, measure the interaction among them and finally synthesiseall the information to arrive at priorities, which depict preferences [2]. The AHP is able to assist us to obtaining a detailed understanding of the evaluation criteria for the dissertation.

This paper put forward a quantitative method for the evaluation of dissertation for master degree. In order to avoid the randomness, fuzziness, uncertainty/intangible and instability, the AHP is adopted. Multiple evaluation criteria and sub-criteria are organized into a hierarchy. The relative importance of these criteria is accessed and different weight values are assigned to evaluation criteria. As the result, the evaluation process of the dissertation for master degree becomes more simple and convenient. The evaluation result is more objective and exact.

2. Related Work

A handbook for linking levels, learning outcomes and assessment, believes that the use of effective assessment criteria is a long way behind the learning outcomes and level descriptors [3]. Generally the thesis evaluation criterion may be broken into two quantity and quality, which can be applicable to any assessment. No matter which kind of evaluation system is adopted, a set of evaluation criteria should be selected first.

If the evaluation method is qualitative, each review expert is asked to give a short description for each criterion and a grade for the whole thesis such as good, acceptable or unacceptable. Lovitts gives guidelines for reviewer how to allocate a final grade for a dissertation based on qualitative assessment system. The grade may be outstanding, very good, acceptable and unacceptable [4]. It is stated that many dissertation-committee members can recognize a quality dissertation when they see it, adding that they can describe general characteristics of outstanding, very good, acceptable and unacceptable dissertations. Webster et al discussed some common general criteria and their ambiguity for the dissertation assessment [5].

Arefi et al discusses the quality of master’s degree program from the requirements of students. They classifies the quality requirements into four categories of attractive, one-dimensional, must-be and indifferent. Finally, better value and worse value were measured. But none deals with quantitative nature of criteria [6]. Indeed, no fundamental framework for assessment linking assessment items and quantitative criteria has been found in related literature. Woolf points out that there are no explicit evidences indicating whether some criteria are more important than others or markers may weight the criteria [7].

If the evaluation system is quantitative, each criterion is specified by a weight value. Review expert gives a mark for each criterion. The final score for the thesis is calculated according to the weight value specified to the criteria and mark given by reviewer. The evaluation system of thesis should be explicit and consistent. Brits et al focus on the evaluation of the extensive mini-dissertations in the master’s degree in family medicine. Different weight values are specified to 12 assessment criteria and the coefficient of variation was used to compare variations between the different criteria [8]. Freeman and Lewis introduced principles of good assessment criteria mainly in terms of presentation for the dissertation [9]. A hierarchical structure of assessment elements is proposed for thesis assessment weight calculation. Selection principles of thesis assessment components and criteria were discussed as well [10]. Khambadkone select different assessment criteria for different stages of assessment. Different weight values are specified to different stage of assessment as well [11].

The AHP is one of the most commonly used methods of assessment. The AHP handles complex decision-making problem by structuring the complex problem into a simple and comprehensible hierarchical structure. Ji et al, implement the assessment of network security through AHP. The security policies are weighted by a hierarchical structure [12]. Lirn et al adopt AHP to weight the criteria for port selection. These criteria include port efficiency, adequate infrastructure, frequency of ship visits, and so on [13]. AHP also is used to weight the assessment criteria for the project management by Al-Harbi [14].

3. Evaluation System of Dissertation for Master Degree

In China, there are two prevalent kinds of evaluation methods for quality of master thesis. One is qualitative. The other is quantitative. No matter which kind of evaluation system is adopted, a set of evaluation criteria should be selected first. Then if the evaluation method is qualitative, each reviewer is asked to give a short description for each criterion and a grade for the whole thesis such as outstanding, good, acceptable and unacceptable. If the evaluation system is quantitative, each criterion is specified by a weighting value. Reviewer gives a mark for each criterion. The score for the whole thesis is calculated according to the weighting and mark of each criteria.

The quantitative method is more objective than qualitative method. In order to build a quantitative evaluation system for the master thesis two essential issues must be resolved. First of all, a set of criteria must be selected for dissertation evaluation. Then weight values are allocated to corresponding criteria. The criterion with bigger weight value contributes more for the description of the quality for the dissertation.

Evaluation Criteria. Usually, criteria are organized into two levels. First the fundamental criteria such as topic selection, thesis writing and contribution are found. Then each criterion of them is refined into sub-criteria in fine grained. For example, the thesis writing can be refined as content, structure, methodology and normalization.

Weight Coefficient Assignment. Determine the weight coefficient is an important step for the building of the quality evaluation system. To a great extent, weight value for each criterion effects the evaluation of the degree thesis. The rationality and accuracy of the weights directly influence the reliability of the evaluation results. The quality evaluation for the master dissertation can be regarded as a kind of multiple decision problem. The analytic hierarchy process (AHP) is a decision support tool for complex multi-criteria decision problems. This paper adopts AHP to assign weighting values to evaluation criteria.

4. Analytic Hierarchy Process (AHP) Decision Model

"Analytical Hierarchy Process (AHP) is an approach to decision making that involves structuring multiple choice criteria into a hierarchy, assessing the relative importance of these criteria, comparing alternatives for each criterion, and determining an overall ranking of the alternatives" [15]. Applicability, simplicity and low labor-intensity are three obvious advantages of AHP. As introduced before, AHP can be applied for multiple criteria decision making and weight coefficient assignment for criteria. The basic steps of AHP for multiple criteria decision making are as following:

(1) Define the problem and determine its goal.

(2) Structure the hierarchy from the top (the objectives from a decision-maker's viewpoint) through the intermediate levels (criteria on which subsequent levels depend) to the lowest level which usually contains the list of alternatives.

Table 1. Pair-wise comparison scale for AHP preferences [2].

Numerical rating Verbal judgments of preferences
9 Extremely preferred
8 Very strongly to extremely
7 Very strongly preferred
6 Strongly to very strongly
5 Strongly preferred
4 Moderately to strongly
3 Moderately preferred
2 Equally to moderately
1 Equally preferred

(3) Construct a set of pair-wise comparison matrices for each of the lower levels with one matrix for each element in the level immediately above by using the relative scale measurement shown in Table 1. The pair-wise comparisons are done in terms of which element dominates the other.

(4) There are n(n-1)/ judgments required to develop the set of matrices in step (3). Reciprocals are automatically assigned in each pair-wise comparison.

(5) Hierarchical synthesis is now used to weight the eigenvectors by the weights of the criteria and the sum is taken over all weighted eigenvector entries corresponding to those in the next lower level of the hierarchy.

(6) The consistency is determined by using the eigenvalue, λmax, to calculate the consistency index, CI=(λmax-n)/(n-1), where n is the matrix size. Judgment consistency can be checked by taking the consistency ratio (CR) of CI with the appropriate value in Table 2. The CR is acceptable if it is lower than 0.10. Or else the judgment matrix is inconsistent.

Table 2. Average random consistency (RI).

Size of matrix 1 2 3 4 5 6 7 8 9 10
Random consistency 0 0 0.58 0.9 1.12 1.24 1.32 1.41 1.45 1.49

5. AHP Based Weighing Method for Evaluation Criteria

Here AHP is used for assign weight coefficient for evaluation criteria. Figure 1 is the hierarchy built for criteria weighting. The hierarchy is consisted of overall objective and selection criteria. Four fundamental criteria and twelve sub-criteria are selected to evaluate the quality of dissertation for master degree.

Figure 1. Hierachy for criteria weighting.

We compared the criteria and sub-criteria according to their relative importance with respect to the parent element in the adjacent upper level. The judgment matrices for criteria and sub-criteria are given in table 2-table 6. The consistency ratios of those are 0.0000, 0.0000, 0.0687, 0.0688 and 0.0000 respectively. The values of the consistency ratios are all within the acceptable range of 0.10.

Table 3. Judgment matrix of fundamental criteria for objective.

Quality of dissertation for master degree A1 A2 A3 A4
A1 1 1/7 1/5 1/3
A2 7 1 3 5
A3 5 1/3 1 3
A4 3 1/5 1/3 1

Table 4. Judgment matrix of sub-criteria for A1.

A1 B11 B12
B11 1 5
B12 1/5 1

Table 5. Judgment matrix of sub-criteria for A2.

A2 B21 B22 B23 B24
B21 1 2 1/5 1/8
B22 1/2 1 1/4 1/7
B23 5 4 1 4
B24 8 7 1/4 1

Table 6. Judgment matrix of sub-criteria for A3.

A3 B31 B32 B33
B31 1 6 4
B32 1/6 1 3
B33 1/4 1/3 1

Table 7. Judgment matrix of sub-criteria for A4.

A4 B41 B42 B43
B41 1 2 4
B42 1/2 1 3
B43 1/4 1/3 1

The weighting result for the criteria and sub-criteria is listed in table 8.

Table 8. Weight coefficient of criteria and sub-criteria.

Criteria Weight value Sub-criteria Weight value
A1-Topic selection 0.1244 B11-Theoretic foundation 0.0858
B12-Application value 0.0386
A2-Research ability 0.4131 B21-Basic theory and specialized knowledge 0.0502
B22-Ability to work in scientific research 0.0613
B23-Research method 0.1508
B24-Innovation 0.1508
A3-Writing ability 0.2769 B31-Literature review 0.1457
B32-Organization of the thesis 0.0700
B33-Standardization on writing 0.0612
A4-Contribution 0.1856 B41-Economic benefits 0.0784
B42-Social benefits 0.0642
B43-Promotional value 0.0430

6. Case Study and Conclusion

According to the evaluation criteria selected and the weight coefficient assigned by AHP, the score for the whole thesis is calculated according to the weighting and mark of each criteria. As the result, the quality evaluation result of a dissertation for master degree is calculated as: B11×0.0858+B12×0.0386+B21×0.0502+B22×0.0613+B23×0.1508+B24×0.1508+B31×0.1457+B32×0.0700+B33×0.0612+B41×0.0784+B42×0.0642+B43×0.0430.

There are two dissertations, d1 and d2, for master degree. The marks of sub-criteria for d1 are 70, 80, 68, 70, 70, 70, 75, 78, 75, 67, 70 and 68. The marks of sub-criteria for d2 are 85, 78, 83, 78, 82, 87, 75, 81, 78, 77, 70 and 78. According to our method, the quality score for d1 and d2 are 72 (71.5589) and 80 (79.9929) respectively.

From the case study given above, we can find our method makes the assignment of the reviewer simple and clear, the calculating of the quality score for the dissertation objective and fair.

Acknowledgement

This work was supported by Beijing Key Laboratory (NO: BZ0211), Beijing Intelligent Logistics System Collaborative Innovation Center and Reform Project in BWU Postgraduate Education (054160584407).


References

  1. Harvey Woolf. Assessment criteria: reflections on current practices [J]. Assessment & Evaluation in Higher Education, 2004, 29: 4, 479-493.
  2. Saaty TL. How to make a decision: the analytic hierarchy process. European Journal of Operational Research, 1990, 48(1): 9-26.
  3. Moon J, Moon J. The Module and Programme Development Handbook- a practical resource for linking levels, learning outcomes & assessment [J]. University of New South, 2002, Selwood Printing Ltd. Great Briton.
  4. Lovitts B E. How to Grade a Dissertation [J]. Academe, 2005, 91(6):18-23.
  5. Frank Webster, David Pepper, Alan Jenkins. Assessing the Undergraduate Dissertation[J]. Assessment & Evaluation in Higher Education, 2010, 25(1):71-80.
  6. Arefi M, Heidari M, Morkani G S, et al. Application of Kano Model in Higher Education Quality Improvement: Study Master's Degree Program of Educational Psychology in State Universities of Tehran[J]. World Applied Sciences Journal, 2012, 17(3).
  7. Harvey Woolf. Assessment criteria: reflections on current practices[J]. Assessment & Evaluation in Higher Education, 2004, 29(4):479-493.
  8. Brits H, Bezuidenhout J, Steinberg W J, et al. An evaluation of the assessment tool used for extensive mini-dissertations in the Master's Degree in Family Medicine, University of the Free State[J]. Official Journal of the South African Academy of Family Practice/primary Care, 2014, 56(2):125-133.
  9. Freeman R and Lewis R. Planning and implementing assessment, Routledge Falmer, Talylor & Francis group, 1998, London, and New York.
  10. Kim H S. Thesis assessment element weighting and quantitative nature of criterion[J]. Thesis Assessment Criteria, 2009.
  11. Khambadkone A, Kassim A A, Samudra G. Design of Assessment for Project modules with Dissertation Type of session[J]. 2002.
  12. Ji, Xuhua, and C. Pattinson. "AHP Implemented Security Assessment and Security Weight Verification." IEEE Second International Conference on Social Computing IEEE Computer Society, 2010:1026-1031.
  13. Lirn T C, Thanopoulou H A, Beynon M J, et al. An Application of AHP on Transhipment Port Selection: A Global Perspective[M]// Port Management. Palgrave Macmillan UK, 2015.
  14. Al-Harbi A S. Application of the AHP in project management [J]. International Journal of Project Management, 2001, 19(1):19-27.
  15. Power, D. J. Decision Support Systems Glossary, DSSResources.COM, World Wide Web, http://DSSResources.COM/glossary/, 2007. Last updated June 9, 2009.

Article Tools
  Abstract
  PDF(233K)
Follow on us
ADDRESS
Science Publishing Group
548 FASHION AVENUE
NEW YORK, NY 10018
U.S.A.
Tel: (001)347-688-8931