For software testing teachers interested in extending their CS students' experience, we propose an innovative solution that adds two, less traditional-yet fascinating-ingredients: hardware and mistakes. Highlighted here are a few original components, including: (1) miniature, microcontroller-based systems-under-test that mimic real-life IoT devices, such as a smart home; (2) a gentle tutorial on how embedded systems work and fail, and (3) a bug-hunting game to assess students' testing skills. A first deployment to 96 Msc CS students showed that their frequent, unforgettable "Eureka!"moments richly compensate for (limited) accidental hardware damage and (substantial) technical and financial efforts. Future plans include extending the collection of systems and bugs, and sharing the design of this unusual-for-CS intervention with other interested educators. Our hope is that a new generation of professionals will emerge, better prepared to engineer the safe systems we all can rely on.</p

Althuisius, Niels

Limburg, Rob

Molenaar, Mario

Silvis-Cividjian, Natalia

Veltman, Jasper

Visser, Glenn

VU Research Portal

VU Research PortalHouse of the Rising FlamesSilvis-Cividjian, Natalia; Visser, Glenn; Veltman, Jasper; Althuisius, Niels; Limburg, Rob;Molenaar, Mariopublished inITiCSE 20232023DOI (link to publisher)10.1145/3587103.3594153document versionPublisher's PDF, also known as Version of recorddocument licenseArticle 25fa Dutch Copyright ActLink to publication in VU Research Portalcitation for published version (APA)Silvis-Cividjian, N., Visser, G., Veltman, J., Althuisius, N., Limburg, R., & Molenaar, M. (2023). House of theRising Flames: A Hands-on, Bug-centered Tutorial on Embedded Software Testing. In ITiCSE 2023:Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education (pp. 581-582). (Annual Conference on Innovation and Technology in Computer Science Education, ITiCSE; Vol. 2).Association for Computing Machinery. https://doi.org/10.1145/3587103.3594153General rightsCopyright and moral rights for the publications made accessible in the public portal are retained by the authors and/or other copyright ownersand it is a condition of accessing publications that users recognise and abide by the legal requirements associated with these rights.            • Users may download and print one copy of any publication from the public portal for the purpose of private study or research.            • You may not further distribute the material or use it for any profit-making activity or commercial gain            • You may freely distribute the URL identifying the publication in the public portalTake down policyIf you believe that this document breaches copyright please contact us providing details, and we will remove access to the work immediatelyand investigate your claim.E-mail address:vuresearchportal.ub@vu.nlDownload date: 13. Sep. 2024House of the Rising Flames: A Hands-on, Bug-centered Tutorialon Embedded Software TestingNatalia Silvis-CividjianVrije Universiteit (VU)Computer Science DepartmentAmsterdam, The Netherlandsn.silvis-cividjian@vu.nlGlenn VisserJasper VeltmanVrije Universiteit (VU)Amsterdam, The NetherlandsNiels AlthuisiusRob LimburgMario MolenaarVU Bèta ElektronicaAmsterdam, The NetherlandsABSTRACTFor software testing teachers interested in extending their CS stu-dents’ experience, we propose an innovative solution that addstwo, less traditional-yet fascinating-ingredients: hardware and mis-takes. Highlighted here are a few original components, including:(1) miniature, microcontroller-based systems-under-test that mimicreal-life IoT devices, such as a smart home; (2) a gentle tutorialon how embedded systems work and fail, and (3) a bug-huntinggame to assess students’ testing skills. A first deployment to 96 MscCS students showed that their frequent, unforgettable “Eureka!”moments richly compensate for (limited) accidental hardware dam-age and (substantial) technical and financial efforts. Future plansinclude extending the collection of systems and bugs, and sharingthe design of this unusual-for-CS intervention with other interestededucators. Our hope is that a new generation of professionals willemerge, better prepared to engineer the safe systems we all canrely on.CCS CONCEPTS• Software and its engineering; • Computer systems organi-zation → Embedded and cyber-physical systems;KEYWORDSsoftware testing education, embedded systems, IoT, fault injectionACM Reference Format:Natalia Silvis-Cividjian, Glenn Visser, Jasper Veltman, Niels Althuisius, RobLimburg, and Mario Molenaar. 2023. House of the Rising Flames: A Hands-on, Bug-centered Tutorial on Embedded Software Testing . In Proceedingsof the 2023 Conference on Innovation and Technology in Computer ScienceEducation V. 2 (ITiCSE 2023), July 8–12, 2023, Turku, Finland. ACM, NewYork, NY, USA, 2 pages. https://doi.org/10.1145/3587103.35941531 THE PROBLEMDedicated software testing courses in CS curricula are rare. More-over, the few existing courses [2, 5, 7, 10] teach mostly unit testingof standalone and correct software artifacts, such as ATMs, webservers, or library management systems. Unfortunately, this lim-ited experience does not match an IoT-dominated reality, wherePermission to make digital or hard copies of part or all of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for third-party components of this work must be honored.For all other uses, contact the owner/author(s).ITiCSE 2023, July 8–12, 2023, Turku, Finland© 2023 Copyright held by the owner/author(s).ACM ISBN 979-8-4007-0139-9/23/07.https://doi.org/10.1145/3587103.3594153software is embedded in smart systems and once in a while causesaccidents that make the headlines.2 A SOLUTIONIn an attempt to close the gap, we designed a 30h tutorial on em-bedded systems testing, as part of an existing 156h introductorysoftware testing course, read at the Vrije Universiteit (VU) in Am-sterdam. The tutorial is a product emerged from VU-BugZoo, aproject that aims to innovate the way we teach software testing[11, 12]. First we had to solve typical challenges related to embed-ded testing, such as (1) deciding which systems-under-test anddiagnostic & emulation equipment to use, and (2) managing therisk of equipment damage, which is much higher when hardwareis present [4]. The result is a tutorial suitable for CS students withbasic knowledge of standalone software testing that is innovativein three ways. First, it is hands-on. Students learn in a lab how tocode and test embedded software for realistic toy IoT systems, suchas a thermostat and a self-driving car. The C++ software-under-testis embedded in the FRDM multifunctional control unit, based ona NXP-FRDM-K66F microcontroller [8] (see Fig. 1). All artifactsdescribed here are designed and built in-house. Next, the tutorialis bug-centered. Along with "sunny-day" scenarios, students alsoexperience how embedded systems can fail. The failures vary froma LED that fails to turn on because of a wrong label in the code("D1" instead of "D0"), to an overheated resistor because of a wrongparameter (10 instead of 0.1). Students are encouraged to drawparallels with real-life situations, such as an airbag failing to openduring a crash, or a neonatal incubator heater catching fire. Finally,the tutorial includes a gamified assessment, in which students huntfor hidden bugs in VU-SmartHome, la pièce de résistance of thisproject. VU-SmartHome combines Lego blocks with software andsensing/actuating electronics using an original design, illustratedin Fig. 1. Examples of injected bugs are: "The garage door openstoo late", or "When it is dark outside, the curtains open instead ofclosing". Different from similar gamified initiatives [1, 3], we arerewarding a sound test strategy above the number of bugs found.3 FIRST RESULTSThe intervention was deployed and evaluated with 96 CS students.At the beginning of the course, around 60% of students had abso-lutely no previous experience with hardware or diagnostic equip-ment. After the tutorial completion, students learned how to tweakand extend their existing standalone software testing skills to alsotest embedded software. The possibility to make mistakes and seetheir effects enhanced the learning effect and helped students to581ITiCSE 2023, July 8–12, 2023, Turku, Finland Natalia Silvis-Cividjian et al.Figure 1: Top: Enthusiastic students in the lab (photos takenwith permission); Middle: The FRDM control unit and VU-SmartHome; Bottom: A detailed view of the VU-SmartHome.better understand the causality of notorious accidents, like Therac-25 [6] and Boeing 737 MAX [9]. Moreover, 87% of participantsfound that testing of corrupted embedded software is more excitingthan testing one’s own product, developed with the best intentions.Finally, 78% of participants agreed that embedded testing shouldbe a mandatory component in any software testing course. Theseresults should encourage software testing educators to expose theirstudents to hardware and bugs. The tutorial is a living document,available in GoogleDrive that can support this decision.However, there are also downsides to this out-the-box approach.First, not all students blindly followed us into error. Also, it wasdifficult to create bugs of the same difficulty level for all groups.Moreover, students foundmore bugs in VU-SmartHome, confirmingthat teachers are also humans. Next, students managed to destroymore electronics than we could predict. For example, real smokehas risen with two occasions: from a wrongly connected sensorand from a garage door servomotor which was activated too long.Luckily, the total damage was acceptable. Finally, building the ex-tensive technical infrastructure required a substantial financial andtechnical effort (around 700 euros material costs for a FRDM controlunit and 300 euros for VU-SmartHome). Also, one has to realizethat more costs will follow, as older equipment becomes inevitablyobsolete.4 CONCLUSIONS AND FUTUREWORKWe presented a successful hands-on, bug-centered tutorial on em-bedded software testing. We are open to share our approach withother, software testing or physical computing educators. Our hopeis that a new generation of professionals will emerge, better pre-pared to engineer the high-quality IoT we all can rely on.ACKNOWLEDGMENTSThe VU-BugZoo project is funded by the NRO, The NetherlandsInitiative for Education Research, as part of a Comenius TeachingFellow grant. The authors would like to thank all TAs and studentsparticipating in the 2021-2022 course, for their commitment andLars Eeuwijk, Frans Hendriks, Ramon Parmessar, Joost Rosier, fromthe Fine Mechanics and Engineering group, for their contributionin crafting the complex and innovative technical infrastructure forthis tutorial.REFERENCES[1] Benjamin S. Clegg, JoséMiguel Rojas, and Gordon Fraser. 2017. Teaching SoftwareTesting Concepts Using a Mutation Testing Game (ICSE-SEET ’17). IEEE Press,33–36. https://doi.org/10.1109/ICSE-SEET.2017.1[2] Lee Copeland. 2004. A Practitioner’s Guide to Software Test Design. Artech House.[3] S. Elbaum, S. Person, J. Dokulil, and M. Jorde. 2007. Bug Hunt: Making Early Soft-ware Testing Lessons Engaging and Affordable. In 29th International Conferenceon Software Engineering (ICSE’07). 688–697. https://doi.org/10.1109/ICSE.2007.23[4] Vahid Garousi, Michael Felderer, Çağrı Murat Karapıçak, and Uğur Yılmaz. 2018.Testing embedded software: A survey of the literature. Information and SoftwareTechnology 104 (2018), 14–45. https://doi.org/10.1016/j.infsof.2018.06.016[5] Paul C. Jorgensen. 2013. Software Testing: a Craftsman’s Approach. AuerbachPublications.[6] N.G. Leveson and C.S. Turner. 1993. An investigation of the Therac-25 accidents.Computer 26, 7 (1993), 18–41. https://doi.org/10.1109/MC.1993.274940[7] Aditya Mathur. 2008. Foundations of Software Testing. Addison-Wesley.[8] NXP. [n.d.]. https://www.nxp.com/design/development-boards/freedom-development-boards/mcu-boards/freedom-development-platform-for-kinetis-k66-k65-and-k26-mcus:FRDM-K66F. Accessed: 2023-11-3.[9] Online. [n.d.]. Boeing 737 MAX: What’s Happened After the 2 DeadlyCrashes. https://www.nytimes.com/interactive/2019/business/boeing-737-crashes.html. Accessed: 2020-10-10.[10] Mauro Pezzè and Michal Young. 2007. Software testing and analysis - process,principles and techniques. Wiley.[11] Natalia Silvis-Cividjian, Rob Limburg, Niels Althuisius, Emil Apostolov, ViktorBonev, Robert Jansma, Glenn Visser, and Marc Went. 2020. VU-BugZoo: APersuasive Platform for Teaching Software Testing. In Proceedings of the 2020ACM Conference on Innovation and Technology in Computer Science Education.553. https://doi.org/10.1145/3341525.3393975[12] Natalia Silvis-Cividjian, Marc Went, Robert Jansma, Viktor Bonev, and EmilApostolov. 2021. Good Bug Hunting: Inspiring and Motivating Software TestingNovices (ITiCSE ’21). Association for Computing Machinery, New York, NY, USA,171–177. https://doi.org/10.1145/3430665.3456330582

House of the Rising Flames:A Hands-on, Bug-centered Tutorial on Embedded Software Testing

Abstract

Similar works

Full text

Available Versions

VU Research Portal