Semi-automatic assessment of unrestrained Java code: a Library, a DSL, and a workbench to assess exams and exercises

© ACM 2015. This is the author's version of the work. It is posted here for your personal use. Not for redistribution. The definitive Version of Record was published in http://dx.doi.org/10.1145/2729094.2742615Automated marking of multiple-choice exams is of great interest in university courses with a large number of students. For this reason, it has been systematically implanted in almost all universities. Automatic assessment of source code is however less extended. There are several reasons for that. One reason is that almost all existing systems are based on output comparison with a gold standard. If the output is the expected, the code is correct. Otherwise, it is reported as wrong, even if there is only one typo in the code. Moreover, why it is wrong remains a mystery. In general, assessment tools treat the code as a black box, and they only assess the externally observable behavior. In this work we introduce a new code assessment method that also verifies properties of the code, thus allowing to mark the code even if it is only partially correct. We also report about the use of this system in a real university context, showing that the system automatically assesses around 50% of the work.This work has been partially supported by the EU (FEDER) and the Spanish Ministerio de Economíay Competitividad (Secretaría de Estado de Investigación, Desarrollo e Innovación) under grant TIN2013-44742-C4-1-R and by the Generalitat Valenciana under grant PROMETEOII2015/013. David Insa was partially supported by the Spanish Ministerio de Educación under FPU grant AP2010-4415.Insa Cabrera, D.; Silva, J. (2015). Semi-automatic assessment of unrestrained Java code: a Library, a DSL, and a workbench to assess exams and exercises. ACM. https://doi.org/10.1145/2729094.2742615SK. A Rahman and M. Jan Nordin. A review on the static analysis approach in the automated programming assessment systems. In National Conference on Programming 07, 2007.K. Ala-Mutka. A survey of automated assessment approaches for programming assignments. In Computer Science Education, volume 15, pages 83--102, 2005.C. Beierle, M. Kula, and M. Widera. Automatic analysis of programming assignments. In Proc. der 1. E-Learning Fachtagung Informatik (DeLFI '03), volume P-37, pages 144--153, 2003.J. Biggs and C. Tang. Teaching for Quality Learning at University : What the Student Does (3rd Edition). In Open University Press, 2007.P. Denny, A. Luxton-Reilly, E. Tempero, and J. Hendrickx. CodeWrite: Supporting student-driven practice of java. In Proceedings of the 42nd ACM technical symposium on Computer science education, pages 09--12, 2011.R. Hendriks. Automatic exam correction. 2012.P. Ihantola, T. Ahoniemi, V. Karavirta, and O. Seppala. Review of recent systems for automatic assessment of programming assignments. In Proceedings of the 10th Koli Calling International Conference on Computing Education Research, pages 86--93, 2010.H. Kitaya and U. Inoue. An online automated scoring system for Java programming assignments. In International Journal of Information and Education Technology, volume 6, pages 275--279, 2014.M.-J. Laakso, T. Salakoski, A. Korhonen, and L. Malmi. Automatic assessment of exercises for algorithms and data structures - a case study with TRAKLA2. In Proceedings of Kolin Kolistelut/Koli Calling - Fourth Finnish/Baltic Sea Conference on Computer Science Education, pages 28--36, 2004.Y. Liang, Q. Liu, J. Xu, and D. Wang. The recent development of automated programming assessment. In Computational Intelligence and Software Engineering, pages 1--5, 2009.K. A. Naudé, J. H. Greyling, and D. Vogts. Marking student programs using graph similarity. In Computers & Education, volume 54, pages 545--561, 2010.A. Pears, S. Seidman, C. Eney, P. Kinnunen, and L. Malmi. Constructing a core literature for computing education research. In SIGCSE Bulletin, volume 37, pages 152--161, 2005.F. Prados, I. Boada, J. Soler, and J. Poch. Automatic generation and correction of technical exercices. In International Conference on Engineering and Computer Education (ICECE 2005), 2005.M. Supic, K. Brkic, T. Hrkac, Z. Mihajlovic, and Z. Kalafatic. Automatic recognition of handwritten corrections for multiple-choice exam answer sheets. In Information and Communication Technology, Electronics and Microelectronics (MIPRO), pages 1136--1141, 2014.S. Tung, T. Lin, and Y. Lin. An exercise management system for teaching programming. In Journal of Software, 2013.T. Wang, X. Su, Y. Wang, and P. Ma. Semantic similarity-based grading of student programs. In Information and Software Technology, volume 49, pages 99--107, 2007

Real-time agreement and fulfilment of SLAs in Cloud Computing environments

A Cloud Computing system must readjust its resources by taking into account the demand for its services. This raises the need for designing protocols that provide the individual components of the Cloud architecture with the ability to self-adapt and to reach agreements in order to deal with changes in the services demand. Furthermore, if the Cloud provider has signed a Service Level Agreement (SLA) with the clients of the services that it offers, the appropriate agreement mechanism has to ensure the provision of the service contracted within a specified time. This paper introduces real-time mechanisms for the agreement and fulfilment of SLAs in Cloud Computing environments. On the one hand, it presents a negotiation protocol inspired by the standard WSAgreement used in web services to manage the interactions between the client and the Cloud provider to agree the terms of the SLA of a service. On the other hand, it proposes the application of a real-time argumentation framework for redistributing resources and ensuring the fulfilment of these SLAs during peaks in the service demand.This work is supported by the Spanish government Grants CONSOLIDER-INGENIO 2010 CSD2007-00022, TIN2011-27652-C03-01, TIN2012-36586-C03-01 and TIN2012-36586-C03-03.De La Prieta, F.; Heras Barberá, SM.; Palanca Cámara, J.; Rodríguez, S.; Bajo, J.; Julian Inglada, VJ. (2014). Real-time agreement and fulfilment of SLAs in Cloud Computing environments. AI Communications. 1-24. doi:10.3233/AIC-140626S124[1]V. Aleven and K.D. Ashley, Teaching case-based argumentation through a model and examples, empirical evaluation of an intelligent learning environment, in: Artificial Intelligence in Education, AIED-97, Frontiers in Artificial Intelligence and Applications, Vol. 39, IOS Press, 1997, pp. 87–94.[2]M. Alhamad, W. Perth, T. Dillon and E. Chang, Conceptual SLA framework for cloud computing, in: 4th IEEE International Conference on Digital Ecosystems and Technologies (DEST), IEEE Press, 2010, pp. 606–610.Armbrust, M., Stoica, I., Zaharia, M., Fox, A., Griffith, R., Joseph, A. D., … Rabkin, A. (2010). A view of cloud computing. Communications of the ACM, 53(4), 50. doi:10.1145/1721654.1721672Ashley, K. D. (1991). Reasoning with cases and hypotheticals in HYPO. International Journal of Man-Machine Studies, 34(6), 753-796. doi:10.1016/0020-7373(91)90011-u[6]P. Barham, B. Dragovic, K. Fraser, S. Hand, T. Harris, A. Ho, R. Neugebauer, I. Pratt and A. Warfield, Xen and the art of virtualization, in: 9th ACM Symposium on Operating Systems Principles (SOSP-03), ACM Press, 2003, pp. 164–177.Beloglazov, A., Abawajy, J., & Buyya, R. (2012). Energy-aware resource allocation heuristics for efficient management of data centers for Cloud computing. Future Generation Computer Systems, 28(5), 755-768. doi:10.1016/j.future.2011.04.017[8]A. Beloglazov and R. Buyya, Energy efficient allocation of virtual machines in cloud data centers, in: 10th IEEE/ACM International Conference on Cluster, Cloud and Grid Computing, IEEE Computer Society, 2010, pp. 577–578.[9]A. Beloglazov and R. Buyya, Energy efficient resource management in virtualized cloud data centers, in: 10th IEEE/ACM International Conference on Cluster, Cloud and Grid Computing, IEEE Computer Society, 2010, pp. 826–831.Bench-Capon, T., & Sartor, G. (2003). A model of legal reasoning with cases incorporating theories and values. Artificial Intelligence, 150(1-2), 97-143. doi:10.1016/s0004-3702(03)00108-5[11]T.J. Bench-Capon, Specification and implementation of Toulmin dialogue game, in: International Conferences on Legal Knowledge and Information Systems, JURIX-98, Frontiers of Artificial Intelligence and Applications, IOS Press, 1998, pp. 5–20.[12]R. Buyya, R. Ranjan and R.N. Calheiros, Intercloud: Utility-oriented federation of cloud computing environments for scaling of application services, in: 10th International Conference on Algorithms and Architectures for Parallel Processing – Volume Part I, ICA3PP’10, Springer-Verlag, 2010, pp. 13–31.[13]R. Buyya, C.S. Yeo and S. Venugopal, Market-oriented cloud computing: Vision, hype, and reality for delivering it services as computing utilities, in: High Performance Computing and Communications, 2008. HPCC’08. 10th IEEE International Conference, September 2008, IEEE, 2008, pp. 5–13.Chen, C., Li, S. S., Chen, B., & Wen, D. (2011). Agent Recommendation for Agent-Based Urban-Transportation Systems. IEEE Intelligent Systems, 26(6), 77-81. doi:10.1109/mis.2011.94[15]Y.Y. Cheng, M. Low, S. Zhou, W. Cai and C.S. Choo, Evolving agent-based simulations in the clouds, in: 3rd International Workshop on Advanced Computational Intelligence (IWACI), 2010, pp. 244–249.[16]F. Dignum and H. Weigand, Communication and Deontic Logic, in: Information Systems – Correctness and Reusability. Selected Papers from the IS-CORE Workshop, R. Wieringa and R. Feenstra, eds, World Scientific Publishing Co., 1995, pp. 242–260.Erdogmus, H. (2009). Cloud Computing: Does Nirvana Hide behind the Nebula? IEEE Software, 26(2), 4-6. doi:10.1109/ms.2009.31[19]J.O. Fitó, I. Goiri and J. Guitart, SLA-driven elastic cloud hosting provider, in: 18th Euromicro International Conference on Parallel, Distributed and Network-Based Processing (PDP), IEEE Computer Society, 2010, pp. 111–118.Fuentes-Fernández, R., Hassan, S., Pavón, J., Galán, J. M., & López-Paredes, A. (2012). Metamodels for role-driven agent-based modelling. Computational and Mathematical Organization Theory, 18(1), 91-112. doi:10.1007/s10588-012-9110-5Heras, S., Botti, V., & Julián, V. (2009). Challenges for a CBR framework for argumentation in open MAS. The Knowledge Engineering Review, 24(4), 327-352. doi:10.1017/s0269888909990178Heras, S., Jordán, J., Botti, V., & Julián, V. (2013). Argue to agree: A case-based argumentation approach. International Journal of Approximate Reasoning, 54(1), 82-108. doi:10.1016/j.ijar.2012.06.005[24]M. Jensen, J. Schwenk, N. Gruschka and L. Iacono, On technical security issues in cloud computing, in: IEEE International Conference on Cloud Computing, IEEE Press, 2009, pp. 109–116.Kakas, A., Maudet, N., & Moraitis, P. (2005). Modular Representation of Agent Interaction Rules through Argumentation. Autonomous Agents and Multi-Agent Systems, 11(2), 189-206. doi:10.1007/s10458-005-2176-4[26]M.J. Kim, H.G. Yoon and H.K. Lee, MAV: An intelligent Multi-agent model based on Cloud computing for resource virtualization, in: Computers, Networks, Systems, and Industrial Engineering, Studies in Computational Intelligence, Vol. 365, Springer, 2011, pp. 99–111.Kraus, S., Sycara, K., & Evenchik, A. (1998). Reaching agreements through argumentation: a logical model and implementation. Artificial Intelligence, 104(1-2), 1-69. doi:10.1016/s0004-3702(98)00078-2[28]W.-Y. Lin, G.-Y. Lin and H.-Y. Wei, Dynamic auction mechanism for cloud resource allocation, in: 10th IEEE/ACM International Conference on Cluster, Cloud and Grid Computing, CCGRID’10, IEEE Computer Society, Washington, DC, USA, 2010, pp. 591–592.[29]S. Liu, G. Quan and S. Ren, On-line scheduling of real-time services for cloud computing, in: 6th World Congress on Services, SERVICES’10, IEEE Computer Society, 2010, pp. 459–464.Navarro, M., Heras, S., Botti, V., & Julián, V. (2013). Towards real-time agreements. Expert Systems with Applications, 40(10), 3906-3917. doi:10.1016/j.eswa.2012.12.087Ontañón, S., & Plaza, E. (2011). An argumentation framework for learning, information exchange, and joint-deliberation in multi-agent systems1. Multiagent and Grid Systems, 7(2-3), 95-108. doi:10.3233/mgs-2011-0169Palanca, J., Navarro, M., García-Fornes, A., & Julian, V. (2013). Deadline prediction scheduling based on benefits. Future Generation Computer Systems, 29(1), 61-73. doi:10.1016/j.future.2012.05.007[33]C. Pautasso, O. Zimmermann and F. Leymann, Restful web services vs. “big”’ web services: making the right architectural decision, in: Proceedings of the 17th International Conference on World Wide Web, WWW’08, ACM, New York, NY, USA, 2008, pp. 805–814.[34]J. Peng, X. Zhang, Z. Lei, B. Zhang, W. Zhang and Q. Li, Comparison of several cloud computing platforms, in: 2nd International Symposium on Information Science and Engineering, ISISE’09, IEEE Computer Society, 2009, pp. 23–27.Prakken, H., & Sartor, G. (1998). Artificial Intelligence and Law, 6(2/4), 231-287. doi:10.1023/a:1008278309945[36]I. Rahwan and G. Simari, eds, Argumentation in Artificial Intelligence, Springer, 2009.Ross, J. W., & Westerman, G. (2004). Preparing for utility computing: The role of IT architecture and relationship management. IBM Systems Journal, 43(1), 5-19. doi:10.1147/sj.431.0005Schaffer, H. E. (2009). X as a Service, Cloud Computing, and the Need for Good Judgment. IT Professional, 11(5), 4-5. doi:10.1109/mitp.2009.112[39]K.M. Sim, Agent-based cloud commerce, in: IEEE International Conference on Industrial Engineering and Engineering Management, IEEE Press, 2009, pp. 717–721.Soh, L.-K., & Tsatsoulis, C. (2005). A Real-Time Negotiation Model and A Multi-Agent Sensor Network Implementation. Autonomous Agents and Multi-Agent Systems, 11(3), 215-271. doi:10.1007/s10458-005-0539-5Talia, D. (2012). Clouds Meet Agents: Toward Intelligent Cloud Services. IEEE Internet Computing, 16(2), 78-81. doi:10.1109/mic.2012.28Tolchinsky, P., Modgil, S., Atkinson, K., McBurney, P., & Cortés, U. (2011). Deliberation dialogues for reasoning about safety critical actions. Autonomous Agents and Multi-Agent Systems, 25(2), 209-259. doi:10.1007/s10458-011-9174-5[44]A. Toniolo, T. Norman and K. Sycara, An empirical study of argumentation schemes in deliberative dialogue, in: 20th European Conference on Artificial Intelligence, ECAI-12, Frontiers in Artificial Intelligence and Applications, Vol. 242, IOS Press, 2012, pp. 756–761.[45]W.-T. Tsai, Q. Shao, X. Sun and J. Elston, Real-time service-oriented cloud computing, in: IEEE 6th World Congress on Services, SERVICES’10, IEEE Press, 2010, pp. 473–478.[46]D. Walton, C. Reed and F. Macagno, Argumentation Schemes, Cambridge University Press, 2008.[47]L. Wang, J. Tao, M. Kunze, A. Castellanos, D. Kramer and W. Karl, Scientific cloud computing: Early definition and experience, in: 10th IEEE International Conference on High Performance Computing and Communications (HPCC-08), IEEE Press, 2008, pp. 825–830.[48]Y.O. Yazir, C. Matthews, R. Farahbod, S. Neville, A. Guitouni, S. Ganti and Y. Coady, Dynamic resource allocation in computing clouds using distributed multiple criteria decision analysis, in: IEEE 3rd International Conference on Cloud Computing (CLOUD), IEEE Computer Society, 2010, pp. 91–98.[49]Y. Yu, S. Ren, N. Chen and X. Wang, Profit and penalty aware (pp-aware) scheduling for tasks with variable task execution time, in: ACM Symposium on Applied Computing, SAC’10, ACM, 2010, pp. 334–339

A précis of philosophy of computing and information technology

The authors recently finished a comprehensive chapter on “Philosophy of Computing and Information Technology” for the forthcoming (fall 2009) Philosophy of Technology and Engineering Sciences (Ed.: A. Meijers), Volume IX in the Elsevier series Handbook of the Philosophy of Science (Eds.: D. Gabbay, P. Thagard and J. Woods). The purpose of the chapter is to review and discuss the main developments, concepts, topics, and contributors in the intersection between philosophy and computing, as well as provide some suggestions on how to structure the many subcategories within what is loosely referred to as philosophy of computing. In this short synopsis, we will give an outline of the kinds of issues raised in this chapter

Route Planning in Transportation Networks

We survey recent advances in algorithms for route planning in transportation networks. For road networks, we show that one can compute driving directions in milliseconds or less even at continental scale. A variety of techniques provide different trade-offs between preprocessing effort, space requirements, and query time. Some algorithms can answer queries in a fraction of a microsecond, while others can deal efficiently with real-time traffic. Journey planning on public transportation systems, although conceptually similar, is a significantly harder problem due to its inherent time-dependent and multicriteria nature. Although exact algorithms are fast enough for interactive queries on metropolitan transit systems, dealing with continent-sized instances requires simplifications or heavy preprocessing. The multimodal route planning problem, which seeks journeys combining schedule-based transportation (buses, trains) with unrestricted modes (walking, driving), is even harder, relying on approximate solutions even for metropolitan inputs.Comment: This is an updated version of the technical report MSR-TR-2014-4, previously published by Microsoft Research. This work was mostly done while the authors Daniel Delling, Andrew Goldberg, and Renato F. Werneck were at Microsoft Research Silicon Valle