Melodic, using music to train visually impaired kids in computational thinking

Dissertação de mestrado em Engenharia InformáticaThis document, in context of second year of Integrated Master of Informatics Engineering, reports the development of a project that intends to teach Computational Thinking to kids with special educational needs, in this case blindness. The aim of this research is to characterize both subjects, Computational Thinking and Blindness, and identify what are the current most used and best practises to teach this different way of thinking to kids with special needs. To achieve this, Melodic was created. This is a system composed by a software and a hardware where the user must create sequences with the tactile blocks (the hardware) and then read them with the mobile application (the software), that converts the sequence created into sound. With this, the user can easily hear the differences that the changes in the blocks sequence can make. This can be compared to the Computational Thinking teaching through the use of robots, because in that case, users can see the result of their instructions in the robot movement and with Melodic, the user can hear the result of their instruction with the musical note sequence played by the app. In this document more technical aspects such as the architecture of the application that is proposed to accomplish the goal of the present Master’s project, will also be discussed. After this, the project development process that lead to the creation of Melodic is described as well as all the decisions taken. A description of all functionalities of this system can also be seen in this document. To prove the research hypothesis initially stated, some exercises were created and described. The referred exercises were designed to access if Melodic actually develops Computational Thinking.Este documento, no contexto do segundo ano do Mestrado Integrado em Engenharia Informática, relata o desenvolvimento de um projecto que pretende ensinar Pensamento Computacional a crianças com necessidades educativas especiais, neste caso a cegueira. O objetivo desta investigação é caraterizar tanto o Pensamento Computacional como a Cegueira, e identificar quais são atualmente as práticas mais usadas para ensinar esta forma diferente de pensar a crianças com necessidades especiais. Para o conseguir, foi criado o Melodic. Este é um sistema composto por software e hardware onde o utilizador deve criar sequências com os blocos tácteis (o hardware) e depois lê-los com a aplicação móvel (o software), que converte a sequência criada em som. Com isto, o utilizador pode facilmente ouvir as diferenças que as alterações na sequência dos blocos podem fazer. Isto pode ser comparado com o ensino Pensamento Computacional através do uso de robôs, sendo que nesse caso os utilizadores podem ver o resultado das suas instruções no movimento do robô e com Melodic, os utilizadores podem ouvir o resultado das suas instruções com a sequência de notas musicais tocadas pela aplicação. Este é um sistema composto por um software e um hardware onde o utilizador deve criar sequências com os blocos tácteis (o hardware) e depois lê-los com a aplicação móvel (o software), que converte a sequência criada em som. Neste documento serão também discutidos aspetos mais técnicos, tais como a arquitetura da aplicação que é proposta para atingir o objectivo deste projecto de mestrado. Depois disto, descreve-se o processo de desenvolvimento do projecto que levou à criação da Melodic, bem como todas as decisões tomadas. Uma descrição de todas as funcionalidades deste sistema também pode ser vista neste documento. Para comprovar a hipótese de investigação inicialmente referida, foram criados e descritos alguns exercícios. Os referidos exercícios foram concebidos para verificar se o Melodic realmente se treina o Pensamento Computacional

Review: Development and technical design of tangible user interfaces in wide-field areas of application

A tangible user interface or TUI connects physical objects and digital interfaces. It is more interactive and interesting for users than a classic graphic user interface. This article presents a descriptive overview of TUI's real-world applications sorted into ten main application areas-teaching of traditional subjects, medicine and psychology, programming, database development, music and arts, modeling of 3D objects, modeling in architecture, literature and storytelling, adjustable TUI solutions, and commercial TUI smart toys. The paper focuses on TUI's technical solutions and a description of technical constructions that influences the applicability of TUIs in the real world. Based on the review, the technical concept was divided into two main approaches: the sensory technical concept and technology based on a computer vision algorithm. The sensory technical concept is processed to use wireless technology, sensors, and feedback possibilities in TUI applications. The image processing approach is processed to a marker and markerless approach for object recognition, the use of cameras, and the use of computer vision platforms for TUI applications.Web of Science2113art. no. 425

Robots for inclusive classrooms: a scoping review

Robot-based activities have been proven to be a valuable tool for children with learning and developmental disabilities. However, their feasibility in general educational environments needs further exploration. This scoping review provides a critical examination of robot-based learning experiences involving children with disabilities, implemented either in mainstream schools or in specialized centers in order to gain insight into their potential to support inclusion. For this purpose, a search was conducted in the multidisciplinary Scopus and WoS databases, completed with Dialnet database. Based on PRISMA guidelines for literature reviews, we limited the systematic analysis to 33 papers published after 2009 that contain information on the instructional design and details of how the activities were implemented. On the other hand, studies reporting interventions with robots for clinical purposes were excluded as well as papers focused exclusively on technical developments. Content analysis shows that most experiences lead to improvements in terms of educational goals and/or stakeholders’ satisfaction. However, the analysis also reported issues that may hinder the adoption of these practices in general classrooms and integrated education services. The reported difficulties include the lack of stability and autonomy of the robots used, the need for aids and adaptations to enable children with sensory and physical impairments to interact easily with the robots, and the requirement of technical support with system’s setup, implementation and maintenance. We conclude that robots and robotics are a powerful tool to address the needs of diverse learners who are included in mainstream classrooms. This review aims at presenting evidences of good practices and recommendations for successful implementation.Open Access funding provided thanks to the CRUE-CSIC agreement with Springer Nature. The authors did not receive support from any organization for the submitted work.Peer ReviewedPostprint (published version

Tangible user interfaces : past, present and future directions

In the last two decades, Tangible User Interfaces (TUIs) have emerged as a new interface type that interlinks the digital and physical worlds. Drawing upon users' knowledge and skills of interaction with the real non-digital world, TUIs show a potential to enhance the way in which people interact with and leverage digital information. However, TUI research is still in its infancy and extensive research is required in or- der to fully understand the implications of tangible user interfaces, to develop technologies that further bridge the digital and the physical, and to guide TUI design with empirical knowledge. This paper examines the existing body of work on Tangible User In- terfaces. We start by sketching the history of tangible user interfaces, examining the intellectual origins of this field. We then present TUIs in a broader context, survey application domains, and review frame- works and taxonomies. We also discuss conceptual foundations of TUIs including perspectives from cognitive sciences, phycology, and philoso- phy. Methods and technologies for designing, building, and evaluating TUIs are also addressed. Finally, we discuss the strengths and limita- tions of TUIs and chart directions for future research

Development of Tangible Code Blocks for the Blind and Visually Impaired

The fields of Science, Technology, Engineering, and Mathematics (STEM) have been growing at an accelerating rate in recent times. Knowing how to program has become one key skill for entering all of these STEM fields. However, many students find programming difficult. The block based programming language, Scratch, was specifically designed to lower hurdles to learning how to program for sighted students. Unfortunately, although very effective and widely used in K12 classrooms, Scratch, similar to other block based languages, is inaccessible to students who are blind and visually impaired (BVI). This thesis is part of a larger project to make the Scratch environment accessible to BVI students. The focus of this thesis is on creating a tangible code block design that: 1) is accessible to BVIs, 2) retains the reduced need to struggle with syntax of Scratch, 3) allows code construction through action, 4) and co-construction with other BVI and sighted students, and 5) can create moderately sized programs at low cost. The first several parts of this thesis consider the design and assessment process for the code blocks, which went through two iterations. The four major components of the first design iteration were: 1) the use of passive blocks, with use of 2) the local edge shape connectivity between blocks defining the program syntax, 3) telescoping tubing to allow nested expressions when valid, and 4) haptically legible commands for both Braille and non-Braille users. The first iteration of the block design was compared to a text based method in building and correcting operator expressions that included both simple and nested expressions of the arithmetic, relational and logical operators. BVI participants produced correct code significantly more when doing the tasks with the code blocks than with the text method. Although the text method was faster, it did not account for any additional time that would be needed to identify and change incorrect code before a program could be run. One weakness of the first iteration was that it was difficult for BVI participants to easily determine connectivity between validly connecting code blocks. The second design iteration considered the effect of embedding different degrees of magnetic attraction within the local shape connection to improve identification of the connectivity. It also considered how to represent some commands that had additional restrictions to those found with most of the other code block types. In particular, we considered the use of different “stopper” designs to prevent numeric literals from being placed in the left slot of a “set” command, which could only accept a variable. Results from a set of studies evaluating the ability of BVI participants to identify the connectivity between blocks found that the magnetic attraction within the connection significantly improved accuracy and ease of use, with the stronger magnetic connections preferred. They also found that a stopper design could be used for “exceptions”, with the longer stopper aligned with the local connection preferred. The final part of the thesis examines the use of the code blocks by the targeted population (BVI students in middle school) in a classroom setting within the context of the entire nonvisual interface. To do this, two day code camps were conducted with BVI middle school students, and recorded on video and audio. Qualitative content analysis was used to verify that the students interacted with the system as intended by the code block design. Results suggest that the students did interact with the code blocks as intended by the design, but minor improvements should be made to increase their ease of use. Participants did appear to have a positive experience with the code blocks and the system overall

Using Inclusive Design to Improve the Accessibility of Informal STEM Education, for Children with Visual Impairment

In this research paper, STEM workshops are designed to provide experiences for twenty-five blind and visually impaired children at a summer camp, with STEM activities that are engaging and fun as well as educational. The aspiration is that the participants should have equitable experiences to their peers without visual impairment, so that they may get the same enjoyment from the STEM workshops as any other participants. Another research goal is to investigate the accessibility features of various commercially available robots, and consider the stability of accessibility features as robots are updated and replaced over time. An analytical autoethnographic approach and an Inclusive Design Model are used, which employ the researcher’s experience as a blind person and children’s feedback to inform ongoing design revisions to the Informal STEM Education activities. Children experimented with playing with and programming robotic toys such as a Bee-bot, Cubetto, Cubelets and Lego Mindstorm EV3, using modified mats and building materials. Video recording, group interviews and direct observation were the data collection tools used. Although all of the STEM education tools used in this study required at least some modification to make them more accessible for the participants, the amount of modification needed varied widely. Some tools were nearly accessible out of the box, while others could not easily be made accessible at all. This suggests many avenues for future research into the accessibility of tools for STEM education, especially robots. The inclusive design of some potential STEM education activities which were not tested, for lack of time, are also described

Tangible user interfaces and social interaction in children with autism

Tangible User Interfaces (TUIs) offer the potential for new modes of social interaction for children with Autism Spectrum Conditions (ASC). Familiar objects that are embedded with digital technology may help children with autism understand the actions of others by providing feedback that is logical and predictable. Objects that move, playback sound or create sound – thus repeating programmed effects – offer an exciting way for children to investigate objects and their effects. This thesis presents three studies of children with autism interacting with objects augmented with digital technology. Study one looked at Topobo, a construction toy augmented with kinetic memory. Children played with Topobo in groups of three of either Typically Developing (TD) or ASC children. The children were given a construction task, and were also allowed to play with the construction sets with no task. Topobo in the task condition showed an overall significant effect for more onlooker, cooperative, parallel, and less solitary behaviour. For ASC children significantly less solitary and more parallel behaviour was recorded than other play states. In study two, an Augmented Knights Castle (AKC) playset was presented to children with ASC. The task condition was extended to allow children to configure the playset with sound. A significant effect in a small sample was found for configuration of the AKC, leading to less solitary behaviour, and more cooperative behaviour. Compared to non-digital play, the AKC showed reduction of solitary behaviour because of augmentation. Qualitative analysis showed further differences in learning phase, user content, behaviour oriented to other children, and system responsiveness. Tangible musical blocks (‘d-touch’) in study three focused on the task. TD and ASC children were presented with a guided/non-guided task in pairs, to isolate effects of augmentation. Significant effects were found for an increase in cooperative symbolic play in the guided condition, and more solitary functional play was found in the unguided condition. Qualitative analysis highlighted differences in understanding blocks and block representation, exploratory and expressive play, understanding of shared space and understanding of the system. These studies suggest that the structure of the task conducted with TUIs may be an important factor for children’s use. When the task is undefined, play tends to lose structure and the benefits of TUIs decline. Tangible technology needs to be used in an appropriately structured manner with close coupling (the distance between digital housing and digital effect), and works best when objects are presented in familiar form

Torino: A Tangible Programming Language Inclusive of Children with Visual Disabilities

© 2018, Copyright © 2018 Taylor & Francis Group, LLC. Across the world, policy initiatives are being developed to engage children with computer programming and computational thinking. Diversity and inclusion has been a strong force in this agenda, but children with disabilities have largely been omitted from the conversation. Currently, there are no age appropriate tools for teaching programming concepts and computational thinking to primary school children with visual disabilities. We address this gap through presenting the design and implementation of Torino, a tangible programming language for teaching programming concepts to children age 7–11 regardless of level of vision. In this paper, we: (1) describe the design process done in conjunction with children with visual disabilities; (2) articulate the design decisions made; and (3) report insights generated from an evaluation with 10 children with mixed visual abilities that considers how children are able to trace (read) and create (write) programs with Torino. We discuss key design trade-offs: (1) readability versus extensibility; and (2) size versus liveness. We conclude by reflecting upon how an inclusive design approach shaped the final result

Tangible programming bricks : an approach to making programming accessible to everyone

Thesis (S.M.)--Massachusetts Institute of Technology, Program in Media Arts & Sciences, February 2000.Includes bibliographical references (leaves 65-68).Thanks to inexpensive microprocessors, consumer electronics are getting more powerful. They offer us greater control over our environment, but in a sense they are getting too powerful for their own good. A programmable thermostat can make my home more comfortable and save energy, but only if I successfully program it to match my life-style. Graphical, direct manipulation user interfaces are step in the direction of making devices easier to program, but it is still easier to manipulate physical objects in the real world than it is to interact with virtual objects "inside" a computer display. Tangible, or graspable user interfaces help bridge the gap between the virtual world and the physical world by allowing us to manipulate digital information directly with our hands. Tangible Programming Bricks are physical building blocks for constructing simple programs. In this thesis I provide technical details of the Bricks themselves, demonstrate that they are useful for controlling a variety of digital "everyday objects," from toy cars to kitchen appliances, and set the stage for future research that will more rigorously support my hypothesis that tangible programming is easier to understand, remember, explain to others, and perform in social settings, when compared to traditional programming mechanisms.by Timothy Scott McNerney.S.M