The automatic assessment of Z specifications

The need to automate the process of assessing a specification in a learning environment is identified to be one of the fundamental ways to improve the use of formal notation in specifying a real system. General issues involved in building an automatic marking system for computer-based courses are explored. Techniques that have been proposed for assessing a specification are also discussed. By considering the issues and the techniques, we describe how they can be used to build a system that is able to give a quality grade to a specification that is written in the Z language. In the system, four quality factors are taken into consideration; maintainability of a specification (which considers the typographic arrangement of a specification and the specification complexity), and correctness of a specification (which reflects the static correctness and the dynamic correctness of a specification). By using suitable quality metrics for specification maintainability, the results that are produced are compared to some values which can either be absolute values or relative to the model answer. The marks awarded for this factor are based on this comparison. Static correctness is carried out by applying a syntax and type checker. The marks granted for this factor depend on the outcome of the checker. Dynamic correctness is determined by employing a testing technique. In the context of a specification, the behaviour of a system-state, which is represented by so-called state variables, is analysed. The specification is 'executed' by using animation. The marks are given according to the correctness of the output and the final state. The system is implemented within the well-known courseware management system, Ceilidh. There are fundamental differences between Z specifications, and the subject matter of other courses taught using the Ceilidh system (which are mostly computer programming courses). For this reason we take some time in this thesis to explain (in some detail) the incorporation of the system within Ceilidh. The need for the fundamental components (i.e the editor, the syntax and type checker, the animator and the automatic marker) are discussed and described. The system has been used by a group of 13 students who attended a Z course within the School of Computer Science and Information Technology at the University of Nottingham during the 1997-1998 academic year. The students were given a questionnaire about the system. An analysis of these questionnaires shows that the currently implemented tools are beneficial and helpful to the students. We also test the results of the system and compare them with a small selected group of human markers. The testing reveals very encouraging results and shows that the system can mark student scripts with a good degree of accuracy. We conclude that this system can provide a very useful aid for teachers of the Z Specification language

The automatic assessment of Z specifications

The need to automate the process of assessing a specification in a learning environment is identified to be one of the fundamental ways to improve the use of formal notation in specifying a real system. General issues involved in building an automatic marking system for computer-based courses are explored. Techniques that have been proposed for assessing a specification are also discussed. By considering the issues and the techniques, we describe how they can be used to build a system that is able to give a quality grade to a specification that is written in the Z language. In the system, four quality factors are taken into consideration; maintainability of a specification (which considers the typographic arrangement of a specification and the specification complexity), and correctness of a specification (which reflects the static correctness and the dynamic correctness of a specification). By using suitable quality metrics for specification maintainability, the results that are produced are compared to some values which can either be absolute values or relative to the model answer. The marks awarded for this factor are based on this comparison. Static correctness is carried out by applying a syntax and type checker. The marks granted for this factor depend on the outcome of the checker. Dynamic correctness is determined by employing a testing technique. In the context of a specification, the behaviour of a system-state, which is represented by so-called state variables, is analysed. The specification is 'executed' by using animation. The marks are given according to the correctness of the output and the final state. The system is implemented within the well-known courseware management system, Ceilidh. There are fundamental differences between Z specifications, and the subject matter of other courses taught using the Ceilidh system (which are mostly computer programming courses). For this reason we take some time in this thesis to explain (in some detail) the incorporation of the system within Ceilidh. The need for the fundamental components (i.e the editor, the syntax and type checker, the animator and the automatic marker) are discussed and described. The system has been used by a group of 13 students who attended a Z course within the School of Computer Science and Information Technology at the University of Nottingham during the 1997-1998 academic year. The students were given a questionnaire about the system. An analysis of these questionnaires shows that the currently implemented tools are beneficial and helpful to the students. We also test the results of the system and compare them with a small selected group of human markers. The testing reveals very encouraging results and shows that the system can mark student scripts with a good degree of accuracy. We conclude that this system can provide a very useful aid for teachers of the Z Specification language

An Analysis on the Applicability of Meta-Heuristic Searching Techniques for Automated Test Data Generation in Automatic Programming Assessment

حظي تقييم البرمجة التلقائي (APA) بالكثير من الاهتمام بين الباحثين بشكل أساسي لدعم الدرجات الآلية ووضع علامات على المهامالمكلف بادائها الطلاب أو التدريبات بشكل منهجي. يتم تعريف APA بشكل شائع كطريقة يمكن أن تعزز الدقة والكفاءة والاتساق وكذلك تقديمملاحظات فورية لحلول للطلاب. في تحقيق APA ، تعد عملية إنشاء بيانات الاختبار مهمة للغاية وذلك لإجراء اختبار ديناميكي لمهمةالطلاب. في مجال اختبار البرمجيات ، أوضحت العديد من الأبحاث التي تركز على توليد بيانات الاختبار نجاح اعتماد تقنيات البحث الفوقية(MHST) من أجل تعزيز إجراءات استنباط بيانات الاختبار المناسبة للاختبار الفعال. ومع ذلك، فإن الأبحاث التي أجريت على APA حتىالآن لم تستغل بعد التقنيات المفيدة لتشمل تغطية اختبار جودة برنامج أفضل. لذلك ، أجرت هذه الدراسة تقييماً مقارنا لتحديد أي تقنية بحثفوقي قابلة للتطبيق لدعم كفاءة توليد بيانات الاختبار الآلي (ATDG) في تنفيذ اختبار وظيفي ديناميكي. في تقييم البرمجة التلقائي يتم تضمينالعديد من تقنيات البحث الفوقية الحديثة في التقييم المقارن الذي يجمع بين كل من خوارزميات البحث المحلية والعالمية من عام 2000 حتىعام 2018 .تشير نتيجة هذه الدراسة إلى أن تهجين Cuckoo Search مع Tabu Search و lévy flight كواحدة من طرق البحث الفوقية الواعدةليتم تطبيقها ، حيث أنه يتفوق على الطرق الفوقية الأخرى فيما يتعلق بعدد التكرارات ونطاق المدخلات.Automatic Programming Assessment (APA) has been gaining lots of attention among researchers mainly to support automated grading and marking of students’ programming assignments or exercises systematically. APA is commonly identified as a method that can enhance accuracy, efficiency and consistency as well as providing instant feedback on students’ programming solutions. In achieving APA, test data generation process is very important so as to perform a dynamic testing on students’ assignment. In software testing field, many researches that focus on test data generation have demonstrated the successful of adoption of Meta-Heuristic Search Techniques (MHST) so as to enhance the procedure of deriving adequate test data for efficient testing. Nonetheless, thus far the researches on APA have not yet usefully exploited the techniques accordingly to include a better quality program testing coverage. Therefore, this study has conducted a comparative evaluation to identify any applicable MHST to support efficient Automated Test Data Generation (ATDG) in executing a dynamic-functional testing in APA. Several recent MHST are included in the comparative evaluation combining both the local and global search algorithms ranging from the year of 2000 until 2018. Result of this study suggests that the hybridization of Cuckoo Search with Tabu Search and lévy flight as one of promising MHST to be applied, as it’s outperforms other MHST with regards to number of iterations and range of inputs

An analysis on the applicability of meta-heuristic searching techniques for automated test data generation in automatic programming assessment

Automatic Programming Assessment (APA) has been gaining lots of attention among researchers mainly to support automated grading and marking of students’ programming assignments or exercises systematically. APA is commonly identified as a method that can enhance accuracy, efficiency and consistency as well as providing instant feedback on students’ programming solutions. In achieving APA, test data generation process is very important so as to perform a dynamic testing on students’ assignment. In software testing field, many researches that focus on test data generation have demonstrated the successful of adoption of Meta-Heuristic Search Techniques (MHST) so as to enhance the procedure of deriving adequate test data for efficient testing. Nonetheless, thus far the researches on APA have not yet usefully exploited the techniques accordingly to include a better quality program testing coverage. Therefore, this study has conducted a comparative evaluation to identify any applicable MHST to support efficient Automated Test Data Generation (ATDG) in executing a dynamic-functional testing in APA. Several recent MHST are included in the comparative evaluation combining both the local and global search algorithms ranging from the year of 2000 until 2018. Result of this study suggests that the hybridization of Cuckoo Search with Tabu Search and lévy flight as one of promising MHST to be applied, as it’s outperforms other MHST with regards to number of iterations and range of inputs

Formative computer based assessment in diagram based domains

This research argues that the formative assessment of student coursework in free-form, diagram-based domains can be automated using CBA techniques in a way which is both feasible and useful. Formative assessment is that form of assessment in which the objective is to assist the process of learning undertaken by the student. The primary deliverable associated with formative assessment is feedback. CBA courseware provides facilities to implement the full lifecycle of an exercise through an integrated, online system. This research demonstrates that CBA offers unique opportunities for student learning through formative assessment, including allowing students to correct their solutions over a larger number of submissions than it would be feasible to allow within the context of traditional assessment forms. The approach to research involves two main phases. The first phase involves designing and implementing an assessment course using the CourseMarker / DATsys CBA system. This system, in common with may other examples of CBA courseware, was intended primarily to conduct summative assessment. The benefits and limitations of the system are identified. The second phase identifies three extensions to the architecture which encapsulate the difference in requirements between summative assessment and formative assessment, presents a design for the extensions, documents their implementation as extensions to the CourseMarker / DATsys architecture and evaluates their contribution. The three extensions are novel extensions for free-form CBA which allow the assessment of the aesthetic layout of student diagrams, the marking of student solutions where multiple model solutions are acceptable and the prioritisation and truncation of feedback prior to its presentation to the student. Evaluation results indicate that the student learning process can be assisted through formative assessment which is automated using CBA courseware. The students learn through an iterative process in which feedback upon a submitted student coursework solution is used by the student to improve their solution, after which they may re-submit and receive further feedback

Formative computer based assessment in diagram based domains

This research argues that the formative assessment of student coursework in free-form, diagram-based domains can be automated using CBA techniques in a way which is both feasible and useful. Formative assessment is that form of assessment in which the objective is to assist the process of learning undertaken by the student. The primary deliverable associated with formative assessment is feedback. CBA courseware provides facilities to implement the full lifecycle of an exercise through an integrated, online system. This research demonstrates that CBA offers unique opportunities for student learning through formative assessment, including allowing students to correct their solutions over a larger number of submissions than it would be feasible to allow within the context of traditional assessment forms. The approach to research involves two main phases. The first phase involves designing and implementing an assessment course using the CourseMarker / DATsys CBA system. This system, in common with may other examples of CBA courseware, was intended primarily to conduct summative assessment. The benefits and limitations of the system are identified. The second phase identifies three extensions to the architecture which encapsulate the difference in requirements between summative assessment and formative assessment, presents a design for the extensions, documents their implementation as extensions to the CourseMarker / DATsys architecture and evaluates their contribution. The three extensions are novel extensions for free-form CBA which allow the assessment of the aesthetic layout of student diagrams, the marking of student solutions where multiple model solutions are acceptable and the prioritisation and truncation of feedback prior to its presentation to the student. Evaluation results indicate that the student learning process can be assisted through formative assessment which is automated using CBA courseware. The students learn through an iterative process in which feedback upon a submitted student coursework solution is used by the student to improve their solution, after which they may re-submit and receive further feedback