Ranking Multi Criteria Decision Making Methods for a Problem by Area Under Receiver Operating Characteristic
Seyed Behnam Khakbaz1, *, Maryam Karimi Davijani2
1Faculty of Management, University of Tehran, Tehran, Iran
2College of Farabi, University of Tehran, Qom, Iran
To cite this article:
Seyed Behnam Khakbaz, Maryam Karimi Davijani. Ranking Multi Criteria Decision Making Methods for a Problem by Area Under Receiver Operating Characteristic.Journal of Investment and Management.Vol.4, No. 5, 2015, pp. 210-215. doi: 10.11648/j.jim.20150405.21
Abstract: One of the major challenges in decision making is selection among MCDM (multi criteria decision making) methods. These methods do not provide same answer to decision maker. Therefore selecting the best answer is an important dilemma. To solve this problem, methods like Borda and Copeland compilation have been proposed. However, applying these methods leads to a hybrid solution which is not necessarily the best answer. In this paper a new approach is proposed to rank different MCDM methods. This approach is AUROC (area under receiver operating characteristic) which is a data mining tool for ranking classification models. The results would show great potential of applying AUROC for ranking MCDM methods in an immense selection problem with historical outcome.
Keywords: Receiver Operating Characteristic, Multi Criteria Decision Making, Area Under ROC, Ranking MADM Methods
There are many multi criteria problems in decision making which have been solved by multi criteria decision making (MCDM) methods. The main goal of MCDM methods is to find the best alternative described by quantitative and qualitative criteria. For this purpose decision makers (DMs) need to determine criteria preference and then MCDM methods that weight the criteria and evaluate the alternatives. MCDM methods are applied for both decision making and technology foresight .
Because of MCDM methods variety, decision makers (DMs) face with a dilemma in many cases, so they need a solution to choose one of these methods.
Up to now, a few researches have studied to compare and choose among MCDM methods. One of these researches is done to develop a conceptual framework to articulate tentative guidelines and choose an appropriate MCDM method for a problem that a DM face to it. In this article, by use of proposed situation tables, DM can compare the situation of his/her problem by them and then choose the appropriate MCDM method . Another solution for this purpose is statistical solutions. In this solution, after ranking the alternatives by MCDM methods, DM can use statistical coefficients (for example Spearman's rank correlation coefficient or Kendall's coefficient of concordance), and choose the best MCDM method for that problem . However, some of problems which use MCDM methods, have a little statistical significant, that DM can't choose the best MCDM method for the problem . Moreover, E. Triantaphyllue et al. proposed a systematic solution to choose the best MCDM method for a problem . In 2007  compared SAW, TOPSIS and VIKOR methods for their problem conditions, and ranked them. In another research, Opricovic and Tzeng compared the TOPSIS and VIKOR methods with their principles . Antucheviciene compared three MCDM methods (TOPSIS, COPRAS and VIKOR) by statistical approach . However, as we can see, these approaches have deficiencies in a general compartment.
As we know, a similar problem is in classification which is selecting best classifier for a problem. In data mining, this problem has been solved by methods like Receiver Operating Characteristic (ROC) curves . Therefore, this approach has been proposed for applying in MCDM problems in this article.
This method (ROC) apply for four different MCDM methods (SAW, ELECTRE, TOPSIS and VIKOR) for a marketing decision making problem by historical data. Then AUROC is used to rank these methods. In the next section, literature of ROC and then four MCDM methods would describe. Afterward research methodology and then findings of this article would explain in the latter section.
2. Literature Review
In the beginning of this section, literature of ROC and its analyzing method would be explained. Afterwards, a brief description of four different MCDM methods (SAW, ELECTRE, TOPSIS and VIKOR) which have been applied for the problem, is expressed.
The ROC (Receiver Operating Characteristic) curves was first used for signal detection in the World War II, in the war a noisy channel should be refined, and because of this purpose, ROC curves were used for signal detection to characterize the tradeoff between true and false alarms. Also ROC curves have widely been used in data mining. By this tool, data miners can compare the performance of different models (classifier models) , . ROC curves depict the performance of a classifier regardless to class distribution or error costs . Using ROC curves, TPR (True Positive Rate) versus FPR (False Positive Rate) are plotted and the best classifier model by comparing shape of the curve and AUROC is chosen. , .
ROC curves demonstrate the performance of models and compare them by drawing a graph in a two dimensional space. This space consists of TPR and FPR. TPR is the percentage of TP (True Positive) over the positives (Equation 1), and FPR is the percentage of FP (False Positive) over the negatives (Equation 2).
Positives are the alternatives or tuples (like an alternative, a tuple is a row in a dataset that have attributes or criterion that should be classify by a classifier) that are actuall target, and negatives are the alternatives or tuples that in the reality are non-target.
TP is the alternatives or tuples which is predicted target (for example those customers who are determined to accept the marketing plan), and actually is the target. FP is also the alternatives or tuples which is predicted in targeting, but they are not actually target. In the table 1, TP and FP are shown.
The area under ROC (AUROC) is a critical issue to select the best model. AUROC is a value between 0.5 and 1, the method that has the closest AUROC to 1, is the best method. For example the graph in the left side of Figure 1 has better efficiency than the graph in the right side , , [i].
Classifier can be binary (for example: for a marketing problem, the customers who accept the marketing plan and they, who do not) or multi classes (for example for a marketing plan that the company proposed three different marketing plans to its customers). Binary classifications are very similar to decision making problems. In decision making, DMs have to accept or reject the alternatives. Correspondingly, this paper propose the use of ROC curve for MCDM problems to choose the best MCDM method for a historical marketing plan problem.
The numerical example of this paper is to choose among customers of a bank, to distinguish those who accept the marketing plan.
2.2. SAW Method
The SAW (Simple Additive Weighted) method was first introduced by Harsanyi in 1955. Because of simple procedure, SAW method has gotten popular, and widely been used in MCDM problems .
Step by step: SAW method:
1. Normalizing decision matrix.
2. Multiplying weight of criteria by normal value of each criteria of every alternatives.
3. Sum up the values created in the last step, and make the point of each alternatives.
4. Choose the alternative that has the maximum point.
2.3. ELECTRE Method
The ELECTRE method (in English: elimination and choice translating reality) was first introduced by Benayoun, et al. in 1966. This method is related to paired comparison of different alternatives. In the end, ELECTRE rank the alternatives by their priorities over others and choose the best of them. ELECTRE's alternatives comparison is based on comparing each criterion for every paired alternative . DMs can use ELECTRE for the problems that have a large number of alternatives and eliminate less favorable ones of them, then because of limited alternatives, DMs can choose the best of them .
Step by step: ELECTRE method:
1. Normalizing decision matrix.
2. Multiplying weight of criteria by normal value of each criterion of every alternative.
3. Determining concordance or discordance of each criterion for every paired comparison of different alternatives.
4. Calculating concordance matrix by summing up the weight of each concordance in paired comparison.
5. Calculating discordance matrix by dividing maximum weight of each discordance in paired comparison by maximum weight of every paired comparison.
6. Determining effective coordinated matrix by concordance matrix.
7. Determining effective uncoordinated matrix by discordance matrix.
8. Calculating effective matrix by multiplying each array of effective coordinated and uncoordinated matrixes.
9. Choose the alternative that has the maximum point in the effective matrix.
2.4. TOPSIS Method
TOPSIS (Technique for Order of Preference by Similarity to Ideal Solution) is a multi criteria decision analysis method, which was originally developed by Hwang and Yoon ,  and . TOPSIS is based on closest Euclidean distance to ideal point and longest distance to nadir point. DMs should specify cost and profit criterions when using TOPSIS for the MCDM problems. TOPSIS is a compensatory method that make trade-offs between poor and good resulted criterions.
Step by step: TOPSIS method:
1. Normalizing decision matrix.
2. Multiplying weight of criteria by normal value of each criterion of every alternative.
3. Determining the ideal and nadir point for every criterion.
4. Calculate the Euclidean distance from ideal and nadir alternative for each alternative.
5. Calculate the relative closeness to ideal alternative.
6. Elect the alternative with the highest relative closeness.
2.5. VIKOR Method
The VIKOR (Vise Kriterijumska Optimizacija I Kompromisno Resenje) method focuses on ranking some alternatives based on some criterions. This method was established by Yu and Zeleny. This method determines a ranking list and a solution by introducing the multi-criteria ranking index based on the particular measure of closeness to the ideal solution .
Steps of this method are:
1. Determine the best and worst values of all criterion functions. If the criterion is a benefit type then the best is max and the worst is min of the function.
2. Compute Si (the maximum group utility) and Ri (the minimum individual regret of the opponent) values.
3. Compute the value Qi by the relation between Si and Ri.
4. Sort the alternatives by the S, R and Q in a decreasing order and rank them.
5. An alternative is ranked the best by the minimum of Q under two below conditions:
• Acceptable advantage
• Acceptable stability in decision making
To draw a ROC curve for a MCDM historical problem, this research amends the original ROC method. In the beginning all alternatives are ranked in a decreasing list by one of the MCDM method. then, next the value of positives and negatives in the historical data must be calculated (for example for a marketing problem, it must be a dataset that have historical data about the previous marketing plan with the response of customers to it, and the criterions that described each of them). By determining the positives, negatives and also ranking list of alternatives, from the first alternative in the ranking, this procedure would follow:
1. If the alternative is a target:
TPR= TPR+ (1/P)
2. If the alternative is not a target:
FPR= FPR + (1/N)
3. Specify the point (FPR, TPR) in the two dimensional graph.
4. If the list is not empty return to 1.
5. Draw the curve that passes through the specified points (drawing ROC curve).
After following these procedures, the ROC curve is ready. Then the area under curve is calculated, so as to calculating AUROC. ROC curve can develop good criterion to compare MCDM methods. For example the method that goes up faster is better, in the case that the cost of surveying is high; however AUROC of this method is lower than other methods, the method preceding the vertical up border is the best method.
In this paper to show the potential of using ROC and AUROC in comparing MCDM methods, a marketing campaign has been applied. The dataset which is used for this purpose is a bank dataset that have about 1000 customers’ data. Each customer is an alternative that it must decide on investing on that customer or not. Therefore, MCDM methods have been used to decide on customers. Fortunately the response of customers to the marketing plan is available; therefore ROC and AUROC can be used to compare MCDM methods and choose the best of them for the future marketing campaign. Each customer is described by nominal, ordinal and numerical criterions, which ordinal criterions are converted to numeric scale by their importance, and each nominal criterion is converted to binary criterions (for example a nominal criterion by three different class convert to three binary criterions). Furthermore, MATLAB 2012 has been used as an analyzing tool.
In this section the analyzing results of applying different MCDM methods for this problem is demonstrated.
4.1. VIKOR Method
For the marketing plan problem, VIKOR is the weakest method, the AUROC for this method is only 0.6046, slightly over 0.5 (Random Predictor that accidentally predicts the target and has the equivalent probability for TPR and FPR). Because of the AUROC's weak result, it can be concluded that VOKOR is a weak method for this problem.
As it can be seen in the (Figure 2), the ROC curve of VIKOR method (the green curve) is very close to the blue straight line (Random Predictor), and it shows the weak result of this method (As it can be seen that in the (FPR= 0.2, TPR= 0.4) point the VIKOR has the best result). Furthermore, VIKOR has more computational complexity cost than SAW and TOPSIS methods. Therefore, this method is an inappropriate method for this problem.
4.2. SAW Method
For this marketing campaign problem, SAW is the second method. The AUROC for this method is 0.6891. This method has better result compare with VIKOR; as a result, SAW is a better method. In addition the AUROC of this method is very close to TOPSIS method which would discuss in the next subsection.
As it can be seen in the (Figure 3), the ROC curve of SAW method (the green curve), has better shape compare with ROC curve of VIKOR method. Moreover, in the (FPR= 0.2, TPR= 0.4) point (best point of SAW), the gradient of SAW and VIKOR ROC are different (they both have equivalent best point). SAW has a very moderate decreasing gradient, compare with VIKOR. Furthermore, the low computational complexity cost of SAW method is a positive point to select it for this problem; it can be the first choice, if the dataset is very huge.
4.3. TOPSIS Method
AUROC of TOPSIS method for this marketing campaign problem is 0.7011. This is the best AUROC of MCDM methods which have been applied for this problem. Therefore, TOPSIS is the best choice for this problem.
As it can be seen in Figure 4, ROC curve for TOPSIS method has a good result in the middle of the curve, which has long distance from Random Predictor (the blue straight line). The best point of ROC curve for TOPSIS method is (FPR=0.4, TPR= 0.8); however the overall AUROC for this method is higher than the SAW method. Choosing between them depends on situation, because the SAW method reaches to the best point sooner, but TOPSIS's best point covers 80% of customers who might accept the marketing plan. Moreover, TOPSIS's computational complexity cost is appropriate for large database.
4.4. Comparing MCDM Methods
To conclude the comparing of MCDM methods in this article, the methods have been compared in one smaller dataset, in six different conditions. The dataset that is used in this section are sampled from original dataset and have information of 100 customers. The six conditions are:
1. Limited criterions- numerical only: in this condition, dataset has only numerical criterions to compare MCDM methods.
2. Limited criterions- nominal only: in this condition, dataset has only the nominal criterions that have selected by feature selection method.
3. Limited criterions- ordinal only: in this condition, dataset has only ordinal criterions to compare MCDM methods.
4. Limited criterions- blend: in this condition, dataset has all the criterions that are selected by feature selection method.
5. Only nominal: in this condition, dataset has all of the nominal criterions to compare the MCDM methods.
6. Complete criterions: in this condition, dataset has all of the criterions of the original dataset.
The result of analyzing is as below:
SAW: This method's AUROC and thus efficiency has dropped as the number of alternatives increase; however remains a good method. Furthermore SAW method has a good result in facing with dataset which consist of numerical or selecting criterions. Totally SAW is a very good method for these problems (marketing campaign).
TOPSIS: TOPSIS is the best MCDM method for the marketing campaign; however the result of TOPSIS has fallen when the number of alternatives increases or when facing with nominal and ordinal criterions.
ELECTRE: ELECTRE method has a good performance; however, because of high computational complexity cost, it can be used for the dataset that has a limited number of alternatives,. In addition ELECTRE has limitations for facing with similar alternatives (for example alternatives which have only nominal or ordinal criterions). Therefore this method is good for the marketing campaign problems, which have limited number of alternatives with blend criterions.
VIKOR: According to analysis, VIKOR is a good method for the problems that have limited alternatives. Also VIKOR is weak to face with nominal and ordinal criterions, thus this method is proposed only for marketing campaign problems which have a small dataset.
The analyzing results have been summarized in the (Table 2).
|Limited criterions||Non limited criterions|
With regards to the contents of previous sections, ROC curves and AUROC are appropriate tools for evaluating MCDM methods. Furthermore, the numerical example of this article has shown that the best MCDM method (between SAW, TOPSIS, ELECTRE and VIKOR) for a marketing campaign problem is TOPSIS; however the SAW method has shown better results when only a section of whole dataset is considered in the problem, because of sooner best point of this method. SAW and VIKOR have the equivalent best point, but because of better AUROC of SAW from beginning point to the best point, SAW is better and has the preference to select. This shows another use of ROC curves to evaluate MCDM methods.
This paper shows AUROC is a good tool for overall evaluating of different methods, and the shape of ROC curve can result in better comparison of them. Applying this tool can be consequences to the conditions of different problems and preferences. For example despite.
TOPSIS has the best result for this article’s marketing plan, when the goal is to select only limited customers in the conditions of problem, with regards to ROC curves of MCDM methods, it can be concluded that the SAW method maybe is the best method for the conditions of this particular problem.
At the end of this article, Table 3 has shown the summery of comparing five MCDM methods (SAW, TOPSIS, ELECTRE, PROMETHEE and VIKOR) that are used for analyzing this paper's numerical example, and compared by their ROC curves and AUROC values.
|Method||Numerous alternatives||Few alternatives||Nominal- ordinal criterions||Complete criterions||Selected criterions by feature selection|
[i] The ROC method that described in this paper is a bit different from the original method in the references, because of condition of the paper.