Kwok, Ron Chi-Wai

English

AIS Electronic Library (AISeL)

Association for Information SystemsAIS Electronic Library (AISeL)AMCIS 1996 Proceedings Americas Conference on Information Systems(AMCIS)8-16-1996A Fuzzy Multi - Criteria Decision Model for theAssessment of Projects in Information SystemsRon Chi-Wai KwokCity University of Hong KongFollow this and additional works at: http://aisel.aisnet.org/amcis1996This material is brought to you by the Americas Conference on Information Systems (AMCIS) at AIS Electronic Library (AISeL). It has been acceptedfor inclusion in AMCIS 1996 Proceedings by an authorized administrator of AIS Electronic Library (AISeL). For more information, please contactelibrary@aisnet.org.Recommended CitationKwok, Ron Chi-Wai, "A Fuzzy Multi - Criteria Decision Model for the Assessment of Projects in Information Systems" (1996). AMCIS1996 Proceedings. 18.http://aisel.aisnet.org/amcis1996/18A Fuzzy Multi-Criteria Decision Model for the Assessment of Projects in Information Systems Ron Chi-Wai Kwok and Jian Ma City University of Hong Kong Introduction Recent studies (Biggs 1989 and Biggs 1992) conducted in several tertiary institutions in Asia show that Asian students are committed to a low level, rote-biased (or surface) approach to learning. The students see the university education as a means to get a degree and to obtain a desirable job based on their extrinsic motivation. They have a tendency to minimise their efforts for avoiding failure. The strategy appropriate to meeting that intention is to limit the target to those essentials that may be reproduced through rote learning.  Assessment of students' learning is a crucial factor that will influence students' approaches to learning. Assessment methods have a strong effect on students' approaches to learning (Elton & Laurillard 1979). Current education research focus on the relationships between different assessment methods and students' approaches to learning (Tompkins & McGraw 1988, Biggs 1992, Ma 1994a, Ma 1994b, Imrie 1995 and Biggs 1996). This paper proposes the use of a new assessment method for assessing the outcomes of students' projects as well as the development and application of a Group Support System (GSS) for supporting the assessment method. In order to provide students with a set of fair and non-threatening assessment requirements, a fuzzy multi-criteria decision model is proposed for the selection of assessment criteria for assessing the project. The GSS creates an electronic environment for lecturers and students to describe their subjective selection of the assessment criteria and the weightings of criteria. The proposed assessment method has been tested in assessing the learning outcomes of students' projects in the Department of Information Systems at the City University of Hong Kong. The Assessment Method The new assessment method focuses on group decision making between students and lecturers through electronic meetings. A Group Support System (GSS) is used to assist the processes for discussing, negotiating, formulating, re-negotiating and revising assessment criteria among students and lecturers.   The conceptual model of the new assessment method is shown in Fig. 1. The tasks can be carried out in the following eight stages: Stage 1: The lecturers design and draft the list of assessment criteria based on the project objectives, the available resources and the assessment guideline and policy of the institution.  Stage 2: Students and lecturers discuss and negotiate, based on their perception of the assessment requirements, the content of the proposed assessment criteria using the GSS. Stage 3: The lecturers then filter the generated opinions and prepare a vote for students and lecturers to select the assessment criteria for the project using the GSS.  Stage 4: Students and lecturers select and rate the assessment criteria with respect to the objectives of the project.  Stage 5: The lecturers then process the result of the vote and formulate the final set of assessment criteria using the GSS.  The final set of the assessment criteria will be announced and the students are allowed to enter their opinions on the results. Using the Delphi method, students and lecturers will then re-vote until most of the group is satisfied with the results, but the final decision is subject to the professional judgement of the lecturers.  Stage 6: Students and lecturers can express their opinion using the GSS on the agreed assessment criteria throughout the duration of the course. The list of the agreed assessment criteria is subject to change if necessary. Students and lecturers can request revision of the assessment.  Stage 7: Lecturers will assess the students' learning outcomes according to the agreed assessment criteria all over the course so that students get continuous feedback from the lecturers, and also students express their opinions on lecturers' comment on their performance.  Stage 8: Students also have the opportunity to self-assess on their own performance and to peer-assess on other students.  Students can input their comment on the performance of their own and of other students. In this way, the lecturers can understand more about how students see themselves, other students and the learning environment so that they can adjust their teaching methods and content, and the learning environment accordingly.  The Fuzzy Criteria Decision Model In order to improve the efficiency and effectiveness of the new assessment method, a GSS is designed to support the selection process of the assessment criteria. A GSS is an interactive computer-based system that facilitates the solution of unstructured problems by a group of decision makers. It is an information technology based environment that supports group meetings, which may be distributed geographically and temporally (Turban 1995). A model of GSS is shown in Figure 2. These components are arranged to support the process of making a decision. The software component of a GSS includes a special decision model base for improvements of the decision making process, a database and an easy-to-use & flexible user interface.  The process of selecting assessment criteria for assessing the students' projects with respect to project's objectives is difficult and time consuming to have the assessment criteria agreed and satisfied by a number of students and lecturers. In order to smoothen the selection process and formulate a set of fair and non-threatening assessment requirements, a fuzzy multi-criteria decision model (Chang & Chen 1994, Grabisch 1995) can be used for assessment criteria selection.   The fuzzy multi-criteria decision model includes:  m alternatives of assessment criteria (A1, A2, ..., Am)  k decision criteria - objectives of the project  (C1, C2,..., Ck)  n decision makers (D1, D2, ..., Dn)  Ritj the rating assigned to alternative Ai under decision criterion Ct by decision maker Dj.  Wtj the weight given to Ct by decision maker Dj.  Fi the final fuzzy score of alternative Ai  where Fi = Ritj * Wtj  The final scores Fi, (i=1, 2,..., m) are ranked to obtain the appropriate alternatives of assessment criteria for the assessment of the project.  For the implementation of the fuzzy multi-criteria decision model, a database schema has been designed as follows:  PERSON(PID, Name);  LECTURER(PID, Appointment, Duty);  STUDENT(PID, Contribution, LearnAppr);  PROJECT(Proj-No, Total-Mark);  ASSIGN(PID, Proj-No);  ASSESSMT CRITERIA(ANo, AssessCriteriaName);  DECISION CRITERIA(CNo, DecisionCriteriaName);  RATING(ANo, CNo, PID, Rating-Score);  WEIGHT(CNo, PID, CNo-Weight);  Figure 3: A database schema for the decision model  The database schema is used to store necessary data for processing decision while the user interface is to allow the group to perform a joint function such as information entry, voting, or ranking alternatives through electronic meetings (Gray, Mandviwalla, Olfman & Satzinger 1993). To the user, the system is the interface. One example of the public screens of the proposed GSS is shown in Fig. 4.   Figure 4: A screen for selecting the assessment criteria  Finally, the goal of the fuzzy multi-criteria decision model, embedded in the GSS, is to aggregate the decision makers' subjective selection of the alternatives of assessment criteria. The agreed assessment criteria are then used to evaluate students' learning outcomes towards the objectives of the project. The proposed GSS improves the productivity and effectiveness of electronic decision making meetings, either by speeding up the decision making process or by improving the quality of the resulting assessment criteria. This is accomplished by providing support to the exchange of ideas, opinions, and preferences within the group. Summary & Further Work This paper proposes a new assessment method for the assessment of projects in Information Systems. It also suggests a fuzzy multi-criteria decision model for selecting assessment criteria. The concept of fuzzy sets is used in this paper because it can easily be used to describe the subjective selection of assessment criteria and the weightings of criteria. However the relative importance of every decision maker is not considered in this paper. New methods of multi-criteria decision making need to be investigated to support the selection of assessment criteria and the weights of criteria from decision makers of different importance. Although the assessment method in this paper is primarily designed to run on the university campus-wide computer network, it can be installed on Internet so that more stakeholders of the assessment policy in higher education, such as parents and future employers etc., can contribute their ideas in the assessment criteria decision processes. The assessment method can be applied for courses running on full-time, part-time or distance learning basis, at educational institutions or business organisations.  References 1. Biggs J.B. (1991), Approaches to Learning in Secondary and Tertiary Students in Hong Kong: Some Comparative Studies, Educational Research Journal, Vol. 6, pp. 27-39.  2. Biggs J.B. (1996), Assessing Learning Quality: reconciling institutional, staff and educational demands, Assessment & Evaluation in Higher Education, Vol. 21, No. 1, pp. 5-15.  3. Biggs J.B. (1989), Students' Approach to Learning in Anglo-Chinese Schools, Educational Research Journal, Vol. 4, pp. 8-17.  4. Biggs J.B. (1992), Why and How do Hong Kong Students Learn? (Using the Learning and Study Process Questionnaires), Faculty of Education HKU, pp. 61-77.  5. Chang P.L. & Chen Y.C. (1994), A fuzzy multi-criteria decision making method for technology transfer strategy selection in biotechnology, Fuzzy Sets and Systems, 63, pp. 131-139.  6. Elton L. & Laurillard D., (1979), Trends in Students Learning, Studies in Higher Education, 4, pp. 87-102.  7. Grabisch M. (1995), Fuzzy Integral in Multicriteria Decision Making, Fuzzy Sets and Systems, 69, pp. 279-298.  8. Gray P., Mandviwalla M., Olfman L. & Satzinger J., (1993), The User Interface in Group Support Systems. In Jessup L. & Valacich J.S., Group Support Systems, Macmillan, pp. 192-213.  9. Ma J., (1994a), Problem-based Learning with Database Systems, Computers & Education, Vol. 22, No. 3, 257-263, Pergamon Press, Oxford, New York.  10. Ma J., (1994b), Problem-based Learning and Students' Approaches in Information Systems, M. Ostwald and A. Kinsland (Eds), Research and Development in Problem-based Learning, Charles Sturt University Press, 101-110.  Acknowledgement The project work has been funded by the 1994 Strategic Research Grant of City University of Hong Kong (project no: 700410) and the 1994/95 Hong Kong Action Learning Project Grant (project code: CITYU23).   

A Fuzzy Multi
-
Criteria Decision Model
for the Assessment of Projects in Information Systems

https://aisel.aisnet.org/cgi/viewcontent.cgi?article=1017&amp;context=amcis1996

A Fuzzy Multi - Criteria Decision Model for the Assessment of Projects in Information Systems

Abstract

Similar works

Full text

Available Versions

AIS Electronic Library (AISeL)