Semi-automatic assessment of unrestrained Java code: a Library, a DSL, and a workbench to assess exams and exercises

© ACM 2015. This is the author's version of the work. It is posted here for your personal use. Not for redistribution. The definitive Version of Record was published in http://dx.doi.org/10.1145/2729094.2742615Automated marking of multiple-choice exams is of great interest in university courses with a large number of students. For this reason, it has been systematically implanted in almost all universities. Automatic assessment of source code is however less extended. There are several reasons for that. One reason is that almost all existing systems are based on output comparison with a gold standard. If the output is the expected, the code is correct. Otherwise, it is reported as wrong, even if there is only one typo in the code. Moreover, why it is wrong remains a mystery. In general, assessment tools treat the code as a black box, and they only assess the externally observable behavior. In this work we introduce a new code assessment method that also verifies properties of the code, thus allowing to mark the code even if it is only partially correct. We also report about the use of this system in a real university context, showing that the system automatically assesses around 50% of the work.This work has been partially supported by the EU (FEDER) and the Spanish Ministerio de Economíay Competitividad (Secretaría de Estado de Investigación, Desarrollo e Innovación) under grant TIN2013-44742-C4-1-R and by the Generalitat Valenciana under grant PROMETEOII2015/013. David Insa was partially supported by the Spanish Ministerio de Educación under FPU grant AP2010-4415.Insa Cabrera, D.; Silva, J. (2015). Semi-automatic assessment of unrestrained Java code: a Library, a DSL, and a workbench to assess exams and exercises. ACM. https://doi.org/10.1145/2729094.2742615SK. A Rahman and M. Jan Nordin. A review on the static analysis approach in the automated programming assessment systems. In National Conference on Programming 07, 2007.K. Ala-Mutka. A survey of automated assessment approaches for programming assignments. In Computer Science Education, volume 15, pages 83--102, 2005.C. Beierle, M. Kula, and M. Widera. Automatic analysis of programming assignments. In Proc. der 1. E-Learning Fachtagung Informatik (DeLFI '03), volume P-37, pages 144--153, 2003.J. Biggs and C. Tang. Teaching for Quality Learning at University : What the Student Does (3rd Edition). In Open University Press, 2007.P. Denny, A. Luxton-Reilly, E. Tempero, and J. Hendrickx. CodeWrite: Supporting student-driven practice of java. In Proceedings of the 42nd ACM technical symposium on Computer science education, pages 09--12, 2011.R. Hendriks. Automatic exam correction. 2012.P. Ihantola, T. Ahoniemi, V. Karavirta, and O. Seppala. Review of recent systems for automatic assessment of programming assignments. In Proceedings of the 10th Koli Calling International Conference on Computing Education Research, pages 86--93, 2010.H. Kitaya and U. Inoue. An online automated scoring system for Java programming assignments. In International Journal of Information and Education Technology, volume 6, pages 275--279, 2014.M.-J. Laakso, T. Salakoski, A. Korhonen, and L. Malmi. Automatic assessment of exercises for algorithms and data structures - a case study with TRAKLA2. In Proceedings of Kolin Kolistelut/Koli Calling - Fourth Finnish/Baltic Sea Conference on Computer Science Education, pages 28--36, 2004.Y. Liang, Q. Liu, J. Xu, and D. Wang. The recent development of automated programming assessment. In Computational Intelligence and Software Engineering, pages 1--5, 2009.K. A. Naudé, J. H. Greyling, and D. Vogts. Marking student programs using graph similarity. In Computers & Education, volume 54, pages 545--561, 2010.A. Pears, S. Seidman, C. Eney, P. Kinnunen, and L. Malmi. Constructing a core literature for computing education research. In SIGCSE Bulletin, volume 37, pages 152--161, 2005.F. Prados, I. Boada, J. Soler, and J. Poch. Automatic generation and correction of technical exercices. In International Conference on Engineering and Computer Education (ICECE 2005), 2005.M. Supic, K. Brkic, T. Hrkac, Z. Mihajlovic, and Z. Kalafatic. Automatic recognition of handwritten corrections for multiple-choice exam answer sheets. In Information and Communication Technology, Electronics and Microelectronics (MIPRO), pages 1136--1141, 2014.S. Tung, T. Lin, and Y. Lin. An exercise management system for teaching programming. In Journal of Software, 2013.T. Wang, X. Su, Y. Wang, and P. Ma. Semantic similarity-based grading of student programs. In Information and Software Technology, volume 49, pages 99--107, 2007

Pattern Recognition and Clustering of Transient Pressure Signals for Burst Location

[EN] A large volume of the water produced for public supply is lost in the systems between sources and consumers. An important-in many cases the greatest-fraction of these losses are physical losses, mainly related to leaks and bursts in pipes and in consumer connections. Fast detection and location of bursts plays an important role in the design of operation strategies for water loss control, since this helps reduce the volume lost from the instant the event occurs until its effective repair (run time). The transient pressure signals caused by bursts contain important information about their location and magnitude, and stamp on any of these events a specific "hydraulic signature". The present work proposes and evaluates three methods to disaggregate transient signals, which are used afterwards to train artificial neural networks (ANNs) to identify burst locations and calculate the leaked flow. In addition, a clustering process is also used to group similar signals, and then train specific ANNs for each group, thus improving both the computational efficiency and the location accuracy. The proposed methods are applied to two real distribution networks, and the results show good accuracy in burst location and characterization.Manzi, D.; Brentan, BM.; Meirelles, G.; Izquierdo Sebastián, J.; Luvizotto Jr., E. (2019). Pattern Recognition and Clustering of Transient Pressure Signals for Burst Location. Water. 11(11):1-13. https://doi.org/10.3390/w11112279S1131111Creaco, E., & Walski, T. (2017). Economic Analysis of Pressure Control for Leakage and Pipe Burst Reduction. Journal of Water Resources Planning and Management, 143(12), 04017074. doi:10.1061/(asce)wr.1943-5452.0000846Campisano, A., Creaco, E., & Modica, C. (2010). RTC of Valves for Leakage Reduction in Water Supply Networks. Journal of Water Resources Planning and Management, 136(1), 138-141. doi:10.1061/(asce)0733-9496(2010)136:1(138)Campisano, A., Modica, C., Reitano, S., Ugarelli, R., & Bagherian, S. (2016). Field-Oriented Methodology for Real-Time Pressure Control to Reduce Leakage in Water Distribution Networks. Journal of Water Resources Planning and Management, 142(12), 04016057. doi:10.1061/(asce)wr.1943-5452.0000697Vítkovský, J. P., Simpson, A. R., & Lambert, M. F. (2000). Leak Detection and Calibration Using Transients and Genetic Algorithms. Journal of Water Resources Planning and Management, 126(4), 262-265. doi:10.1061/(asce)0733-9496(2000)126:4(262)Pérez, R., Puig, V., Pascual, J., Quevedo, J., Landeros, E., & Peralta, A. (2011). Methodology for leakage isolation using pressure sensitivity analysis in water distribution networks. Control Engineering Practice, 19(10), 1157-1167. doi:10.1016/j.conengprac.2011.06.004Jung, D., & Kim, J. (2017). Robust Meter Network for Water Distribution Pipe Burst Detection. Water, 9(11), 820. doi:10.3390/w9110820Colombo, A. F., Lee, P., & Karney, B. W. (2009). A selective literature review of transient-based leak detection methods. Journal of Hydro-environment Research, 2(4), 212-227. doi:10.1016/j.jher.2009.02.003Choi, D., Kim, S.-W., Choi, M.-A., & Geem, Z. (2016). Adaptive Kalman Filter Based on Adjustable Sampling Interval in Burst Detection for Water Distribution System. Water, 8(4), 142. doi:10.3390/w8040142Christodoulou, S. E., Kourti, E., & Agathokleous, A. (2016). Waterloss Detection in Water Distribution Networks using Wavelet Change-Point Detection. Water Resources Management, 31(3), 979-994. doi:10.1007/s11269-016-1558-5Guo, X., Yang, K., & Guo, Y. (2012). Leak detection in pipelines by exclusively frequency domain method. Science China Technological Sciences, 55(3), 743-752. doi:10.1007/s11431-011-4707-3Holloway, M. B., & Hanif Chaudhry, M. (1985). Stability and accuracy of waterhammer analysis. Advances in Water Resources, 8(3), 121-128. doi:10.1016/0309-1708(85)90052-1Sanz, G., Pérez, R., Kapelan, Z., & Savic, D. (2016). Leak Detection and Localization through Demand Components Calibration. Journal of Water Resources Planning and Management, 142(2), 04015057. doi:10.1061/(asce)wr.1943-5452.0000592Zhang, Q., Wu, Z. Y., Zhao, M., Qi, J., Huang, Y., & Zhao, H. (2016). Leakage Zone Identification in Large-Scale Water Distribution Systems Using Multiclass Support Vector Machines. Journal of Water Resources Planning and Management, 142(11), 04016042. doi:10.1061/(asce)wr.1943-5452.0000661Mounce, S. R., & Machell, J. (2006). Burst detection using hydraulic data from water distribution systems with artificial neural networks. Urban Water Journal, 3(1), 21-31. doi:10.1080/15730620600578538Covas, D., Ramos, H., & de Almeida, A. B. (2005). Standing Wave Difference Method for Leak Detection in Pipeline Systems. Journal of Hydraulic Engineering, 131(12), 1106-1116. doi:10.1061/(asce)0733-9429(2005)131:12(1106)Liggett, J. A., & Chen, L. (1994). Inverse Transient Analysis in Pipe Networks. Journal of Hydraulic Engineering, 120(8), 934-955. doi:10.1061/(asce)0733-9429(1994)120:8(934)Caputo, A. C., & Pelagagge, P. M. (2002). An inverse approach for piping networks monitoring. Journal of Loss Prevention in the Process Industries, 15(6), 497-505. doi:10.1016/s0950-4230(02)00036-0Van Zyl, J. E. (2014). Theoretical Modeling of Pressure and Leakage in Water Distribution Systems. Procedia Engineering, 89, 273-277. doi:10.1016/j.proeng.2014.11.187Izquierdo, J., & Iglesias, P. . (2004). Mathematical modelling of hydraulic transients in complex systems. Mathematical and Computer Modelling, 39(4-5), 529-540. doi:10.1016/s0895-7177(04)90524-9Lin, J., Keogh, E., Wei, L., & Lonardi, S. (2007). Experiencing SAX: a novel symbolic representation of time series. Data Mining and Knowledge Discovery, 15(2), 107-144. doi:10.1007/s10618-007-0064-zNavarrete-López, C., Herrera, M., Brentan, B., Luvizotto, E., & Izquierdo, J. (2019). Enhanced Water Demand Analysis via Symbolic Approximation within an Epidemiology-Based Forecasting Framework. Water, 11(2), 246. doi:10.3390/w11020246Meirelles, G., Manzi, D., Brentan, B., Goulart, T., & Luvizotto, E. (2017). Calibration Model for Water Distribution Network Using Pressures Estimated by Artificial Neural Networks. Water Resources Management, 31(13), 4339-4351. doi:10.1007/s11269-017-1750-2Adamowski, J., & Chan, H. F. (2011). A wavelet neural network conjunction model for groundwater level forecasting. Journal of Hydrology, 407(1-4), 28-40. doi:10.1016/j.jhydrol.2011.06.013Brentan, B., Meirelles, G., Luvizotto, E., & Izquierdo, J. (2018). Hybrid SOM+ k -Means clustering to improve planning, operation and management in water distribution systems. Environmental Modelling & Software, 106, 77-88. doi:10.1016/j.envsoft.2018.02.013Calinski, T., & Harabasz, J. (1974). A dendrite method for cluster analysis. Communications in Statistics - Theory and Methods, 3(1), 1-27. doi:10.1080/0361092740882710

Sustainable Higher Education Development through Technology Enhanced Learning

[EN] Higher education is incorporating Information and Communication Technology (ICT) at a fast rate for different purposes. Scientific papers include within the concept of Technology Enhanced Learning (TEL) the myriad applications of information and communication technology, e-resources, and pedagogical approaches to the development of education. TEL¿s specific application to higher education is especially relevant for countries under rapid development for providing quick and sustainable access to quality education (UN sustainable development goal 4). This paper presents the research results of an online pedagogical experience in collaborative academic research for analyzing good practice in TEL-supported higher education development. The results are obtained through a pilot implementation providing curated data on TEL competency¿s development of faculty skills and analysis of developing sustainable higher education degrees through TEL cooperation, for capacity building. Given the increased volume and complexity of the knowledge to be delivered, and the exponential growth of the need for skilled workers in emerging economies, online training is the most effective way of delivering a sustainable higher education. The results of the PETRA Erasmus+ capacity-building project provides evidence of a successful implementation of a TEL-supported methodology for collaborative faculty development focused on future online degrees built collaboratively and applied locally.This research was co-funded by the European Commission through the Erasmus+ KA2 project "Promoting Excellence in Teaching and Learning in Azerbaijani Universities (PETRA)" project number 573630-EPP-1-2016-1-ES-EPPKA2-CBHE-JP.Orozco-Messana, J.; Martínez-Rubio, J.; Gonzálvez-Pons, AM. (2020). Sustainable Higher Education Development through Technology Enhanced Learning. Sustainability. 12(9):1-13. https://doi.org/10.3390/su12093600S113129Abdullah, F., & Ward, R. (2016). Developing a General Extended Technology Acceptance Model for E-Learning (GETAMEL) by analysing commonly used external factors. Computers in Human Behavior, 56, 238-256. doi:10.1016/j.chb.2015.11.036Becker, H. J., & Ravitz, J. (1999). The Influence of Computer and Internet Use on Teachers’ Pedagogical Practices and Perceptions. Journal of Research on Computing in Education, 31(4), 356-384. doi:10.1080/08886504.1999.10782260Mumford, S., & Dikilitaş, K. (2020). Pre-service language teachers reflection development through online interaction in a hybrid learning course. Computers & Education, 144, 103706. doi:10.1016/j.compedu.2019.103706Lee, D., Watson, S. L., & Watson, W. R. (2020). The Relationships Between Self-Efficacy, Task Value, and Self-Regulated Learning Strategies in Massive Open Online Courses. The International Review of Research in Open and Distributed Learning, 21(1), 23-39. doi:10.19173/irrodl.v20i5.4389Passey, D. (2019). Technology‐enhanced learning: Rethinking the term, the concept and its theoretical background. British Journal of Educational Technology, 50(3), 972-986. doi:10.1111/bjet.12783Lai, Y.-C., & Peng, L.-H. (2019). Effective Teaching and Activities of Excellent Teachers for the Sustainable Development of Higher Design Education. Sustainability, 12(1), 28. doi:10.3390/su12010028Lee, S., Lee, H., & Kim, T. (2018). A Study on the Instructor Role in Dealing with Mixed Contents: How It Affects Learner Satisfaction and Retention in e-Learning. Sustainability, 10(3), 850. doi:10.3390/su10030850“Continuous Improvement in Teaching Strategies through Lean Principles”. Teaching & Learning Symposium, University of Southern Indiana http://hdl.handle.net/20.500.12419/455The DeLone and McLean Model of Information Systems Success: A Ten-Year Update. (2003). Journal of Management Information Systems, 19(4), 9-30. doi:10.1080/07421222.2003.11045748Goodman, J., Melkers, J., & Pallais, A. (2019). Can Online Delivery Increase Access to Education? Journal of Labor Economics, 37(1), 1-34. doi:10.1086/698895Alexander, J., Barcellona, M., McLachlan, S., & Sackley, C. (2019). Technology-enhanced learning in physiotherapy education: Student satisfaction and knowledge acquisition of entry-level students in the United Kingdom. Research in Learning Technology, 27(0). doi:10.25304/rlt.v27.2073How Can Adaptive Platforms Improve Student Learning Outcomes? A Case Study of Open Educational Resources and Adaptive Learning Platforms https://ssrn.com/abstract=3478134Sun, A., & Chen, X. (2016). Online Education and Its Effective Practice: A Research Review. Journal of Information Technology Education: Research, 15, 157-190. doi:10.28945/3502EU Commission https://ec.europa.eu/education/education-in-the-eu/digital-education-action-plan_enEssence Project https://husite.nl/essence/Orozco-Messana, J., de la Poza-Plaza, E., & Calabuig-Moreno, R. (2020). Experiences in Transdisciplinary Education for the Sustainable Development of the Built Environment, the ISAlab Workshop. Sustainability, 12(3), 1143. doi:10.3390/su12031143Kurilovas, E., & Kubilinskiene, S. (2020). Lithuanian case study on evaluating suitability, acceptance and use of IT tools by students – An example of applying Technology Enhanced Learning Research methods in Higher Education. Computers in Human Behavior, 107, 106274. doi:10.1016/j.chb.2020.10627

On environment difficulty and discriminating power

The final publication is available at Springer via http://dx.doi.org/10.1007/s10458-014-9257-1This paper presents a way to estimate the difficulty and discriminating power of any task instance. We focus on a very general setting for tasks: interactive (possibly multiagent) environments where an agent acts upon observations and rewards. Instead of analysing the complexity of the environment, the state space or the actions that are performed by the agent, we analyse the performance of a population of agent policies against the task, leading to a distribution that is examined in terms of policy complexity. This distribution is then sliced by the algorithmic complexity of the policy and analysed through several diagrams and indicators. The notion of environment response curve is also introduced, by inverting the performance results into an ability scale. We apply all these concepts, diagrams and indicators to two illustrative problems: a class of agent-populated elementary cellular automata, showing how the difficulty and discriminating power may vary for several environments, and a multiagent system, where agents can become predators or preys, and may need to coordinate. Finally, we discuss how these tools can be applied to characterise (interactive) tasks and (multi-agent) environments. These characterisations can then be used to get more insight about agent performance and to facilitate the development of adaptive tests for the evaluation of agent abilities.I thank the reviewers for their comments, especially those aiming at a clearer connection with the field of multi-agent systems and the suggestion of better approximations for the calculation of the response curves. The implementation of the elementary cellular automata used in the environments is based on the library 'CellularAutomaton' by John Hughes for R [58]. I am grateful to Fernando Soler-Toscano for letting me know about their work [65] on the complexity of 2D objects generated by elementary cellular automata. I would also like to thank David L. Dowe for his comments on a previous version of this paper. This work was supported by the MEC/MINECO projects CONSOLIDER-INGENIO CSD2007-00022 and TIN 2010-21062-C02-02, GVA project PROMETEO/2008/051, the COST - European Cooperation in the field of Scientific and Technical Research IC0801 AT, and the REFRAME project, granted by the European Coordinated Research on Long-term Challenges in Information and Communication Sciences & Technologies ERA-Net (CHIST-ERA), and funded by the Ministerio de Economia y Competitividad in Spain (PCIN-2013-037).José Hernández-Orallo (2015). On environment difficulty and discriminating power. Autonomous Agents and Multi-Agent Systems. 29(3):402-454. https://doi.org/10.1007/s10458-014-9257-1S402454293Anderson, J., Baltes, J., & Cheng, C. T. (2011). Robotics competitions as benchmarks for ai research. The Knowledge Engineering Review, 26(01), 11–17.Andre, D., & Russell, S. J. (2002). State abstraction for programmable reinforcement learning agents. In Proceedings of the National Conference on Artificial Intelligence (pp. 119–125). Menlo Park, CA; Cambridge, MA; London; AAAI Press; MIT Press; 1999.Antunes, L., Fortnow, L., van Melkebeek, D., & Vinodchandran, N. V. (2006). Computational depth: Concept and applications. Theoretical Computer Science, 354(3), 391–404. Foundations of Computation Theory (FCT 2003), 14th Symposium on Fundamentals of Computation Theory 2003.Arai, K., Kaminka, G. A., Frank, I., & Tanaka-Ishii, K. (2003). Performance competitions as research infrastructure: Large scale comparative studies of multi-agent teams. Autonomous Agents and Multi-Agent Systems, 7(1–2), 121–144.Ashcraft, M. H., Donley, R. D., Halas, M. A., & Vakali, M. (1992). Chapter 8 working memory, automaticity, and problem difficulty. In Jamie I.D. Campbell (Ed.), The nature and origins of mathematical skills, volume 91 of advances in psychology (pp. 301–329). North-Holland.Ay, N., Müller, M., & Szkola, A. (2010). Effective complexity and its relation to logical depth. IEEE Transactions on Information Theory, 56(9), 4593–4607.Barch, D. M., Braver, T. S., Nystrom, L. E., Forman, S. D., Noll, D. C., & Cohen, J. D. (1997). Dissociating working memory from task difficulty in human prefrontal cortex. Neuropsychologia, 35(10), 1373–1380.Bordini, R. H., Hübner, J. F., & Wooldridge, M. (2007). Programming multi-agent systems in AgentSpeak using Jason. London: Wiley. com.Boutilier, C., Reiter, R., Soutchanski, M., Thrun, S. et al. (2000). Decision-theoretic, high-level agent programming in the situation calculus. In Proceedings of the National Conference on Artificial Intelligence (pp. 355–362). Menlo Park, CA; Cambridge, MA; London; AAAI Press; MIT Press; 1999.Busoniu, L., Babuska, R., & De Schutter, B. (2008). A comprehensive survey of multiagent reinforcement learning. IEEE Transactions on Systems, Man, and Cybernetics, Part C: Applications and Reviews, 38(2), 156–172.Chaitin, G. J. (1977). Algorithmic information theory. IBM Journal of Research and Development, 21, 350–359.Chedid, F. B. (2010). Sophistication and logical depth revisited. In 2010 IEEE/ACS International Conference on Computer Systems and Applications (AICCSA) (pp. 1–4). IEEE.Cheeseman, P., Kanefsky, B. & Taylor, W. M. (1991). Where the really hard problems are. In Proceedings of IJCAI-1991 (pp. 331–337).Dastani, M. (2008). 2APL: A practical agent programming language. Autonomous Agents and Multi-agent Systems, 16(3), 214–248.Delahaye, J. P. & Zenil, H. (2011). Numerical evaluation of algorithmic complexity for short strings: A glance into the innermost structure of randomness. Applied Mathematics and Computation, 219(1), 63–77Dowe, D. L. (2008). Foreword re C. S. Wallace. Computer Journal, 51(5), 523–560. Christopher Stewart WALLACE (1933–2004) memorial special issue.Dowe, D. L., & Hernández-Orallo, J. (2012). IQ tests are not for machines, yet. Intelligence, 40(2), 77–81.Du, D. Z., & Ko, K. I. (2011). Theory of computational complexity (Vol. 58). London: Wiley-Interscience.Elo, A. E. (1978). The rating of chessplayers, past and present (Vol. 3). London: Batsford.Embretson, S. E., & Reise, S. P. (2000). Item response theory for psychologists. London: Lawrence Erlbaum.Fatès, N. & Chevrier, V. (2010). How important are updating schemes in multi-agent systems? an illustration on a multi-turmite model. In Proceedings of the 9th International Conference on Autonomous Agents and Multiagent Systems: volume 1-Volume 1 (pp. 533–540). International Foundation for Autonomous Agents and Multiagent Systems.Ferber, J. & Müller, J. P. (1996). Influences and reaction: A model of situated multiagent systems. In Proceedings of Second International Conference on Multi-Agent Systems (ICMAS-96) (pp. 72–79).Ferrando, P. J. (2009). Difficulty, discrimination, and information indices in the linear factor analysis model for continuous item responses. Applied Psychological Measurement, 33(1), 9–24.Ferrando, P. J. (2012). Assessing the discriminating power of item and test scores in the linear factor-analysis model. Psicológica, 33, 111–139.Gent, I. P., & Walsh, T. (1994). Easy problems are sometimes hard. Artificial Intelligence, 70(1), 335–345.Gershenson, C. & Fernandez, N. (2012). Complexity and information: Measuring emergence, self-organization, and homeostasis at multiple scales. Complexity, 18(2), 29–44.Gruner, S. (2010). Mobile agent systems and cellular automata. Autonomous Agents and Multi-agent Systems, 20(2), 198–233.Hardman, D. K., & Payne, S. J. (1995). Problem difficulty and response format in syllogistic reasoning. The Quarterly Journal of Experimental Psychology, 48(4), 945–975.He, J., Reeves, C., Witt, C., & Yao, X. (2007). A note on problem difficulty measures in black-box optimization: Classification, realizations and predictability. Evolutionary Computation, 15(4), 435–443.Hernández-Orallo, J. (2000). Beyond the turing test. Journal of Logic Language & Information, 9(4), 447–466.Hernández-Orallo, J. (2000). On the computational measurement of intelligence factors. In A. Meystel (Ed.), Performance metrics for intelligent systems workshop (pp. 1–8). Gaithersburg, MD: National Institute of Standards and Technology.Hernández-Orallo, J. (2000). Thesis: Computational measures of information gain and reinforcement in inference processes. AI Communications, 13(1), 49–50.Hernández-Orallo, J. (2010). A (hopefully) non-biased universal environment class for measuring intelligence of biological and artificial systems. In M. Hutter et al. (Ed.), 3rd International Conference on Artificial General Intelligence (pp. 182–183). Atlantis Press Extended report at http://users.dsic.upv.es/proy/anynt/unbiased.pdf .Hernández-Orallo, J., & Dowe, D. L. (2010). Measuring universal intelligence: Towards an anytime intelligence test. Artificial Intelligence, 174(18), 1508–1539.Hernández-Orallo, J., Dowe, D. L., España-Cubillo, S., Hernández-Lloreda, M. V., & Insa-Cabrera, J. (2011). On more realistic environment distributions for defining, evaluating and developing intelligence. In J. Schmidhuber, K. R. Thórisson, & M. Looks (Eds.), LNAI series on artificial general intelligence 2011 (Vol. 6830, pp. 82–91). Berlin: Springer.Hernández-Orallo, J., Dowe, D. L., & Hernández-Lloreda, M. V. (2014). Universal psychometrics: Measuring cognitive abilities in the machine kingdom. Cognitive Systems Research, 27, 50–74.Hernández-Orallo, J., Insa, J., Dowe, D. L. & Hibbard, B. (2012). Turing tests with turing machines. In A. Voronkov (Ed.), The Alan Turing Centenary Conference, Turing-100, Manchester, 2012, volume 10 of EPiC Series (pp. 140–156).Hernández-Orallo, J. & Minaya-Collado, N. (1998). A formal definition of intelligence based on an intensional variant of Kolmogorov complexity. In Proceedings of International Symposium of Engineering of Intelligent Systems (EIS’98) (pp. 146–163). ICSC Press.Hibbard, B. (2009). Bias and no free lunch in formal measures of intelligence. Journal of Artificial General Intelligence, 1(1), 54–61.Hoos, H. H. (1999). Sat-encodings, search space structure, and local search performance. In 1999 International Joint Conference on Artificial Intelligence (Vol. 16, pp. 296–303).Insa-Cabrera, J., Benacloch-Ayuso, J. L., & Hernández-Orallo, J. (2012). On measuring social intelligence: Experiments on competition and cooperation. In J. Bach, B. Goertzel, & M. Iklé (Eds.), AGI, volume 7716 of lecture notes in computer science (pp. 126–135). Berlin: Springer.Insa-Cabrera, J., Dowe, D. L., España-Cubillo, S., Hernández-Lloreda, M. V., & Hernández-Orallo, J. (2011). Comparing humans and AI agents. In J. Schmidhuber, K. R. Thórisson, & M. Looks (Eds.), LNAI series on artificial general intelligence 2011 (Vol. 6830, pp. 122–132). Berlin: Springer.Knuth, D. E. (1973). Sorting and searching, volume 3 of the art of computer programming. Reading, MA: Addison-Wesley.Kotovsky, K., & Simon, H. A. (1990). What makes some problems really hard: Explorations in the problem space of difficulty. Cognitive Psychology, 22(2), 143–183.Legg, S. (2008). Machine super intelligence. PhD thesis, Department of Informatics, University of Lugano, June 2008.Legg, S., & Hutter, M. (2007). Universal intelligence: A definition of machine intelligence. Minds and Machines, 17(4), 391–444.Leonetti, M. & Iocchi, L. (2010). Improving the performance of complex agent plans through reinforcement learning. In Proceedings of the 2010 International Conference on Autonomous Agents and Multiagent Systems (Vol. 1, pp. 723–730). International Foundation for Autonomous Agents and Multiagent Systems.Levin, L. A. (1973). Universal sequential search problems. Problems of Information Transmission, 9(3), 265–266.Levin, L. A. (1986). Average case complete problems. SIAM Journal on Computing, 15, 285.Li, M., & Vitányi, P. (2008). An introduction to Kolmogorov complexity and its applications (3rd ed.). Berlin: Springer.Low, C. K., Chen, T. Y., & Rónnquist, R. (1999). Automated test case generation for bdi agents. Autonomous Agents and Multi-agent Systems, 2(4), 311–332.Madden, M. G., & Howley, T. (2004). Transfer of experience between reinforcement learning environments with progressive difficulty. Artificial Intelligence Review, 21(3), 375–398.Mellenbergh, G. J. (1994). Generalized linear item response theory. Psychological Bulletin, 115(2), 300.Michel, F. (2004). Formalisme, outils et éléments méthodologiques pour la modélisation et la simulation multi-agents. PhD thesis, Université des sciences et techniques du Languedoc, Montpellier.Miller, G. A. (1956). The magical number seven, plus or minus two: Some limits on our capacity for processing information. Psychological Review, 63(2), 81.Orponen, P., Ko, K. I., Schöning, U., & Watanabe, O. (1994). Instance complexity. Journal of the ACM (JACM), 41(1), 96–121.Simon, H. A., & Kotovsky, K. (1963). Human acquisition of concepts for sequential patterns. Psychological Review, 70(6), 534.Team, R., et al. (2013). R: A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing.Whiteson, S., Tanner, B., & White, A. (2010). The reinforcement learning competitions. The AI Magazine, 31(2), 81–94.Wiering, M., & van Otterlo, M. (Eds.). (2012). Reinforcement learning: State-of-the-art. Berlin: Springer.Wolfram, S. (2002). A new kind of science. Champaign, IL: Wolfram Media.Zatuchna, Z., & Bagnall, A. (2009). Learning mazes with aliasing states: An LCS algorithm with associative perception. Adaptive Behavior, 17(1), 28–57.Zenil, H. (2010). Compression-based investigation of the dynamical properties of cellular automata and other systems. Complex Systems, 19(1), 1–28.Zenil, H. (2011). Une approche expérimentale à la théorie algorithmique de la complexité. PhD thesis, Dissertation in fulfilment of the degree of Doctor in Computer Science, Université de Lille.Zenil, H., Soler-Toscano, F., Delahaye, J. P. & Gauvrit, N. (2012). Two-dimensional kolmogorov complexity and validation of the coding theorem method by compressibility. arXiv, preprint arXiv:1212.6745

Formal Introduction to Fuzzy Implications

SummaryIn the article we present in the Mizar system the catalogue of nine basic fuzzy implications, used especially in the theory of fuzzy sets. This work is a continuation of the development of fuzzy sets in Mizar; it could be used to give a variety of more general operations, and also it could be a good starting point towards the formalization of fuzzy logic (together with t-norms and t-conorms, formalized previously).Institute of Informatics, University of Białystok, PolandMichał Baczyński and Balasubramaniam Jayaram. Fuzzy Implications. Springer Publishing Company, Incorporated, 2008. doi:10.1007/978-3-540-69082-5.Adam Grabowski. Basic formal properties of triangular norms and conorms. Formalized Mathematics, 25(2):93–100, 2017. doi:10.1515/forma-2017-0009.Adam Grabowski. The formal construction of fuzzy numbers. Formalized Mathematics, 22(4):321–327, 2014. doi:10.2478/forma-2014-0032.Adam Grabowski. On the computer certification of fuzzy numbers. In M. Ganzha, L. Maciaszek, and M. Paprzycki, editors, 2013 Federated Conference on Computer Science and Information Systems (FedCSIS), Federated Conference on Computer Science and Information Systems, pages 51–54, 2013.Adam Grabowski. Lattice theory for rough sets – a case study with Mizar. Fundamenta Informaticae, 147(2–3):223–240, 2016. doi:10.3233/FI-2016-1406.Adam Grabowski and Magdalena Jastrzębska. Rough set theory from a math-assistant perspective. In Rough Sets and Intelligent Systems Paradigms, International Conference, RSEISP 2007, Warsaw, Poland, June 28–30, 2007, Proceedings, pages 152–161, 2007. doi:10.1007/978-3-540-73451-2_17.Adam Grabowski and Takashi Mitsuishi. Extending Formal Fuzzy Sets with Triangular Norms and Conorms, volume 642: Advances in Intelligent Systems and Computing, pages 176–187. Springer International Publishing, Cham, 2018. doi:10.1007/978-3-319-66824-6_16.Adam Grabowski and Takashi Mitsuishi. Initial comparison of formal approaches to fuzzy and rough sets. In Leszek Rutkowski, Marcin Korytkowski, Rafal Scherer, Ryszard Tadeusiewicz, Lotfi A. Zadeh, and Jacek M. Zurada, editors, Artificial Intelligence and Soft Computing - 14th International Conference, ICAISC 2015, Zakopane, Poland, June 14-18, 2015, Proceedings, Part I, volume 9119 of Lecture Notes in Computer Science, pages 160–171. Springer, 2015. doi:10.1007/978-3-319-19324-3_15.Adam Grabowski, Artur Korniłowicz, and Adam Naumowicz. Four decades of Mizar. Journal of Automated Reasoning, 55(3):191–198, 2015. doi:10.1007/s10817-015-9345-1.Takashi Mitsuishi, Noboru Endou, and Yasunari Shidama. The concept of fuzzy set and membership function and basic properties of fuzzy set operation. Formalized Mathematics, 9(2):351–356, 2001.Zdzisław Pawlak. Rough sets. International Journal of Parallel Programming, 11:341–356, 1982. doi:10.1007/BF01001956.Lotfi Zadeh. Fuzzy sets. Information and Control, 8(3):338–353, 1965.25324124

Neutrosophic Completion Technique for Incomplete Higher-Order AHP Comparison Matrices

[EN] After the recent establishment of the Sustainable Development Goals and the Agenda 2030, the sustainable design of products in general and infrastructures in particular emerge as a challenging field for the development and application of multicriteria decision-making tools. Sustainability-related decision problems usually involve, by definition, a wide variety in number and nature of conflicting criteria, thus pushing the limits of conventional multicriteria decision-making tools practices. The greater the number of criteria and the more complex the relations existing between them in a decisional problem, the less accurate and certain are the judgments required by usual methods, such as the analytic hierarchy process (AHP). The present paper proposes a neutrosophic AHP completion methodology to reduce the number of judgments required to be emitted by the decision maker. This increases the consistency of their responses, while accounting for uncertainties associated to the fuzziness of human thinking. The method is applied to a sustainable-design problem, resulting in weight estimations that allow for a reduction of up to 22% of the conventionally required comparisons, with an average accuracy below 10% between estimates and the weights resulting from a conventionally completed AHP matrix, and a root mean standard error below 15%.The authors acknowledge the financial support of the Spanish Ministry of Economy and Business, along with FEDER funding (DIMALIFE Project: BIA2017-85098-R).Navarro, IJ.; Martí Albiñana, JV.; Yepes, V. (2021). Neutrosophic Completion Technique for Incomplete Higher-Order AHP Comparison Matrices. Mathematics. 9(5):1-19. https://doi.org/10.3390/math905049611995Worrell, E., Price, L., Martin, N., Hendriks, C., & Meida, L. O. (2001). CARBON DIOXIDE EMISSIONS FROM THE GLOBAL CEMENT INDUSTRY. Annual Review of Energy and the Environment, 26(1), 303-329. doi:10.1146/annurev.energy.26.1.303García, J., Yepes, V., & Martí, J. V. (2020). A Hybrid k-Means Cuckoo Search Algorithm Applied to the Counterfort Retaining Walls Problem. Mathematics, 8(4), 555. doi:10.3390/math8040555Penadés-Plà, V., García-Segura, T., & Yepes, V. (2020). Robust Design Optimization for Low-Cost Concrete Box-Girder Bridge. Mathematics, 8(3), 398. doi:10.3390/math8030398Kim, S., & Frangopol, D. M. (2017). Multi-objective probabilistic optimum monitoring planning considering fatigue damage detection, maintenance, reliability, service life and cost. Structural and Multidisciplinary Optimization, 57(1), 39-54. doi:10.1007/s00158-017-1849-3García-Segura, T., Yepes, V., & Frangopol, D. M. (2017). Multi-objective design of post-tensioned concrete road bridges using artificial neural networks. Structural and Multidisciplinary Optimization, 56(1), 139-150. doi:10.1007/s00158-017-1653-0Van den Heede, P., & De Belie, N. (2014). A service life based global warming potential for high-volume fly ash concrete exposed to carbonation. Construction and Building Materials, 55, 183-193. doi:10.1016/j.conbuildmat.2014.01.033García, J., Martí, J. V., & Yepes, V. (2020). The Buttressed Walls Problem: An Application of a Hybrid Clustering Particle Swarm Optimization Algorithm. Mathematics, 8(6), 862. doi:10.3390/math8060862García-Segura, T., Penadés-Plà, V., & Yepes, V. (2018). Sustainable bridge design by metamodel-assisted multi-objective optimization and decision-making under uncertainty. Journal of Cleaner Production, 202, 904-915. doi:10.1016/j.jclepro.2018.08.177Gursel, A. P., & Ostertag, C. (2016). Comparative life-cycle impact assessment of concrete manufacturing in Singapore. The International Journal of Life Cycle Assessment, 22(2), 237-255. doi:10.1007/s11367-016-1149-yPenadés-Plà, V., Martí, J. V., García-Segura, T., & Yepes, V. (2017). Life-Cycle Assessment: A Comparison between Two Optimal Post-Tensioned Concrete Box-Girder Road Bridges. Sustainability, 9(10), 1864. doi:10.3390/su9101864Navarro, I. J., Yepes, V., & Martí, J. V. (2018). Social life cycle assessment of concrete bridge decks exposed to aggressive environments. Environmental Impact Assessment Review, 72, 50-63. doi:10.1016/j.eiar.2018.05.003Sierra, L. A., Pellicer, E., & Yepes, V. (2017). Method for estimating the social sustainability of infrastructure projects. Environmental Impact Assessment Review, 65, 41-53. doi:10.1016/j.eiar.2017.02.004Navarro, I. J., Yepes, V., & Martí, J. V. (2019). Sustainability assessment of concrete bridge deck designs in coastal environments using neutrosophic criteria weights. Structure and Infrastructure Engineering, 16(7), 949-967. doi:10.1080/15732479.2019.1676791Tavana, M., Shaabani, A., Javier Santos-Arteaga, F., & Raeesi Vanani, I. (2020). A Review of Uncertain Decision-Making Methods in Energy Management Using Text Mining and Data Analytics. Energies, 13(15), 3947. doi:10.3390/en13153947Yannis, G., Kopsacheili, A., Dragomanovits, A., & Petraki, V. (2020). State-of-the-art review on multi-criteria decision-making in the transport sector. Journal of Traffic and Transportation Engineering (English Edition), 7(4), 413-431. doi:10.1016/j.jtte.2020.05.005Navarro, I. J., Penadés-Plà, V., Martínez-Muñoz, D., Rempling, R., & Yepes, V. (2020). LIFE CYCLE SUSTAINABILITY ASSESSMENT FOR MULTI-CRITERIA DECISION MAKING IN BRIDGE DESIGN: A REVIEW. JOURNAL OF CIVIL ENGINEERING AND MANAGEMENT, 26(7), 690-704. doi:10.3846/jcem.2020.13599Hedelin, B. (2018). Complexity is no excuse. Sustainability Science, 14(3), 733-749. doi:10.1007/s11625-018-0635-5Zadeh, L. A. (1973). Outline of a New Approach to the Analysis of Complex Systems and Decision Processes. IEEE Transactions on Systems, Man, and Cybernetics, SMC-3(1), 28-44. doi:10.1109/tsmc.1973.5408575Zadeh, L. A. (1965). Fuzzy sets. Information and Control, 8(3), 338-353. doi:10.1016/s0019-9958(65)90241-xMilošević, D. M., Milošević, M. R., & Simjanović, D. J. (2020). Implementation of Adjusted Fuzzy AHP Method in the Assessment for Reuse of Industrial Buildings. Mathematics, 8(10), 1697. doi:10.3390/math8101697Lin, C.-N. (2020). A Fuzzy Analytic Hierarchy Process-Based Analysis of the Dynamic Sustainable Management Index in Leisure Agriculture. Sustainability, 12(13), 5395. doi:10.3390/su12135395Salehi, S., Jalili Ghazizadeh, M., Tabesh, M., Valadi, S., & Salamati Nia, S. P. (2020). A risk component-based model to determine pipes renewal strategies in water distribution networks. Structure and Infrastructure Engineering, 17(10), 1338-1359. doi:10.1080/15732479.2020.1842466Liu, P., & Liu, X. (2016). The neutrosophic number generalized weighted power averaging operator and its application in multiple attribute group decision making. International Journal of Machine Learning and Cybernetics, 9(2), 347-358. doi:10.1007/s13042-016-0508-0Peng, J., Wang, J., & Yang, W.-E. (2016). A multi-valued neutrosophic qualitative flexible approach based on likelihood for multi-criteria decision-making problems. International Journal of Systems Science, 48(2), 425-435. doi:10.1080/00207721.2016.1218975Saaty, T. L., & Ozdemir, M. S. (2003). Why the magic number seven plus or minus two. Mathematical and Computer Modelling, 38(3-4), 233-244. doi:10.1016/s0895-7177(03)90083-5Harker, P. T. (1987). Incomplete pairwise comparisons in the analytic hierarchy process. Mathematical Modelling, 9(11), 837-848. doi:10.1016/0270-0255(87)90503-3Chen, K., Kou, G., Michael Tarn, J., & Song, Y. (2015). Bridging the gap between missing and inconsistent values in eliciting preference from pairwise comparison matrices. Annals of Operations Research, 235(1), 155-175. doi:10.1007/s10479-015-1997-zBozóki, S., Fülöp, J., & Rónyai, L. (2010). On optimal completion of incomplete pairwise comparison matrices. Mathematical and Computer Modelling, 52(1-2), 318-333. doi:10.1016/j.mcm.2010.02.047Dong, M., Li, S., & Zhang, H. (2015). Approaches to group decision making with incomplete information based on power geometric operators and triangular fuzzy AHP. Expert Systems with Applications, 42(21), 7846-7857. doi:10.1016/j.eswa.2015.06.007Zhou, X., Hu, Y., Deng, Y., Chan, F. T. S., & Ishizaka, A. (2018). A DEMATEL-based completion method for incomplete pairwise comparison matrix in AHP. Annals of Operations Research, 271(2), 1045-1066. doi:10.1007/s10479-018-2769-3Sumathi, I. R., & Antony Crispin Sweety, C. (2019). New approach on differential equation via trapezoidal neutrosophic number. Complex & Intelligent Systems, 5(4), 417-424. doi:10.1007/s40747-019-00117-3Deli, I., & Şubaş, Y. (2016). A ranking method of single valued neutrosophic numbers and its applications to multi-attribute decision making problems. International Journal of Machine Learning and Cybernetics, 8(4), 1309-1322. doi:10.1007/s13042-016-0505-3Ye, J. (2017). Subtraction and Division Operations of Simplified Neutrosophic Sets. Information, 8(2), 51. doi:10.3390/info8020051Liang, R., Wang, J., & Zhang, H. (2017). A multi-criteria decision-making method based on single-valued trapezoidal neutrosophic preference relations with complete weight information. Neural Computing and Applications, 30(11), 3383-3398. doi:10.1007/s00521-017-2925-8Sodenkamp, M. A., Tavana, M., & Di Caprio, D. (2018). An aggregation method for solving group multi-criteria decision-making problems with single-valued neutrosophic sets. Applied Soft Computing, 71, 715-727. doi:10.1016/j.asoc.2018.07.020Sierra, L. A., Pellicer, E., & Yepes, V. (2016). Social Sustainability in the Lifecycle of Chilean Public Infrastructure. Journal of Construction Engineering and Management, 142(5), 05015020. doi:10.1061/(asce)co.1943-7862.0001099Abdel-Basset, M., Manogaran, G., Mohamed, M., & Chilamkurti, N. (2018). RETRACTED: Three-way decisions based on neutrosophic sets and AHP-QFD framework for supplier selection problem. Future Generation Computer Systems, 89, 19-30. doi:10.1016/j.future.2018.06.024Dubois, D. (2011). The role of fuzzy sets in decision sciences: Old techniques and new directions. Fuzzy Sets and Systems, 184(1), 3-28. doi:10.1016/j.fss.2011.06.003Buckley, J. J. (1985). Fuzzy hierarchical analysis. Fuzzy Sets and Systems, 17(3), 233-247. doi:10.1016/0165-0114(85)90090-9Wang, Y.-M., & Elhag, T. M. S. (2006). On the normalization of interval and fuzzy weights. Fuzzy Sets and Systems, 157(18), 2456-2471. doi:10.1016/j.fss.2006.06.008Enea, M., & Piazza, T. (2004). Project Selection by Constrained Fuzzy AHP. Fuzzy Optimization and Decision Making, 3(1), 39-62. doi:10.1023/b:fodm.0000013071.63614.3dChu, T.-C., & Tsao, C.-T. (2002). Ranking fuzzy numbers with an area between the centroid point and original point. Computers & Mathematics with Applications, 43(1-2), 111-117. doi:10.1016/s0898-1221(01)00277-

Algebraic Extensions

In this article we further develop field theory in Mizar [1], [2], [3] towards splitting fields. We deal with algebraic extensions [4], [5]: a field extension E of a field F is algebraic, if every element of E is algebraic over F. We prove amongst others that finite extensions are algebraic and that field extensions generated by a finite set of algebraic elements are finite. From this immediately follows that field extensions generated by roots of a polynomial over F are both finite and algebraic. We also define the field of algebraic elements of E over F and show that this field is an intermediate field of E|F.Christoph Schwarzweller - Institute of Informatics, University of Gdansk, PolandGrzegorz Bancerek, Czesław Bylinski, Adam Grabowski, Artur Korniłowicz, Roman Matuszewski, Adam Naumowicz, Karol Pak, and Josef Urban. Mizar: State-of-the-art and beyond. In Manfred Kerber, Jacques Carette, Cezary Kaliszyk, Florian Rabe, and Volker Sorge, editors, Intelligent Computer Mathematics, volume 9150 of Lecture Notes in Computer Science, pages 261–279. Springer International Publishing, 2015. ISBN 978-3-319-20614-1. doi:10.1007/978-3-319-20615-8 17.Grzegorz Bancerek, Czesław Bylinski, Adam Grabowski, Artur Korniłowicz, Roman Matuszewski, Adam Naumowicz, and Karol Pak. The role of the Mizar Mathematical Library for interactive proof development in Mizar. Journal of Automated Reasoning, 61(1):9–32, 2018. doi:10.1007/s10817-017-9440-6.Adam Grabowski, Artur Korniłowicz, and Christoph Schwarzweller. On algebraic hierarchies in mathematical repository of Mizar. In M. Ganzha, L. Maciaszek, and M. Paprzycki, editors, Proceedings of the 2016 Federated Conference on Computer Science and Information Systems (FedCSIS), volume 8 of Annals of Computer Science and Information Systems, pages 363–371, 2016. doi:10.15439/2016F520.Nathan Jacobson. Basic Algebra I. Dover Books on Mathematics, 1985.Serge Lang. Algebra. Springer, 3rd edition, 2005.Christoph Schwarzweller. Ring and field adjunctions, algebraic elements and minimal polynomials. Formalized Mathematics, 28(3):251–261, 2020. doi:10.2478/forma-2020-0022.291394