Naturalizing a Programming Language via Interactive Learning

Our goal is to create a convenient natural language interface for performing well-specified but complex actions such as analyzing data, manipulating text, and querying databases. However, existing natural language interfaces for such tasks are quite primitive compared to the power one wields with a programming language. To bridge this gap, we start with a core programming language and allow users to "naturalize" the core language incrementally by defining alternative, more natural syntax and increasingly complex concepts in terms of compositions of simpler ones. In a voxel world, we show that a community of users can simultaneously teach a common system a diverse language and use it to build hundreds of complex voxel structures. Over the course of three days, these users went from using only the core language to using the naturalized language in 85.9\% of the last 10K utterances.Comment: 10 pages, ACL201

situ-f: a domain specific language and a first step towards the realization of situ framework

Situ proposes a human centered, dynamic reasoning framework for domain experts to evolve their software. It formally models the relationship between externally observed situation sequences and the rapid evolution of that software system, using real-time usage information from users and contextual capturing on the behavior of a software system relative to its runtime environment. Situf is a continuing effort under Situ framework. In this effort, a domain specific, functional programming language named Situf is presented from its design, semantics and a feasibility test through theoretical validation. The targeted users of this language mainly include domain experts and engineers who are versed in the major concepts and paradigms regarding human-centric situations. As argued there, human-centric situations are vitally important to infer a user\u27s intention and therefore, to drive software service evolution. Situf is designed particularly to encourage domain experts and engineers to think and work with situations. An attribute grammar based approach is developed so that through Situf , relevant real-time contexts can be systematically aggregated around situations. A computational semantics is offered to precisely describe the runtime behavior of a Situf program. While the Situf language serves as the critical centerpiece of this work, its functioning necessarily requires environmental support from Situ elements outside the language itself, such that altogether they give rise to a Situ oriented system. This environment, named Situf -based environment, is also introduced

Ten virtues of structured graphs

This paper extends the invited talk by the first author about the virtues of structured graphs. The motivation behind the talk and this paper relies on our experience on the development of ADR, a formal approach for the design of styleconformant, reconfigurable software systems. ADR is based on hierarchical graphs with interfaces and it has been conceived in the attempt of reconciling software architectures and process calculi by means of graphical methods. We have tried to write an ADR agnostic paper where we raise some drawbacks of flat, unstructured graphs for the design and analysis of software systems and we argue that hierarchical, structured graphs can alleviate such drawbacks

UI Layout Generation with LLMs Guided by UI Grammar

The recent advances in Large Language Models (LLMs) have stimulated interest among researchers and industry professionals, particularly in their application to tasks concerning mobile user interfaces (UIs). This position paper investigates the use of LLMs for UI layout generation. Central to our exploration is the introduction of UI grammar -- a novel approach we proposed to represent the hierarchical structure inherent in UI screens. The aim of this approach is to guide the generative capacities of LLMs more effectively and improve the explainability and controllability of the process. Initial experiments conducted with GPT-4 showed the promising capability of LLMs to produce high-quality user interfaces via in-context learning. Furthermore, our preliminary comparative study suggested the potential of the grammar-based approach in improving the quality of generative results in specific aspects.Comment: ICML 2023 Workshop on AI and HC

Spatially Aware Computing for Natural Interaction

Spatial information refers to the location of an object in a physical or digital world. Besides, it also includes the relative position of an object related to other objects around it. In this dissertation, three systems are designed and developed. All of them apply spatial information in different fields. The ultimate goal is to increase the user friendliness and efficiency in those applications by utilizing spatial information. The first system is a novel Web page data extraction application, which takes advantage of 2D spatial information to discover structured records from a Web page. The extracted information is useful to re-organize the layout of a Web page to fit mobile browsing. The second application utilizes the 3D spatial information of a mobile device within a large paper-based workspace to implement interactive paper that combines the merits of paper documents and mobile devices. This application can overlay digital information on top of a paper document based on the location of a mobile device within a workspace. The third application further integrates 3D space information with sound detection to realize an automatic camera management system. This application automatically controls multiple cameras in a conference room, and creates an engaging video by intelligently switching camera shots among meeting participants based on their activities. Evaluations have been made on all three applications, and the results are promising. In summary, this dissertation comprehensively explores the usage of spatial information in various applications to improve the usability

Improving the translation environment for professional translators

When using computer-aided translation systems in a typical, professional translation workflow, there are several stages at which there is room for improvement. The SCATE (Smart Computer-Aided Translation Environment) project investigated several of these aspects, both from a human-computer interaction point of view, as well as from a purely technological side. This paper describes the SCATE research with respect to improved fuzzy matching, parallel treebanks, the integration of translation memories with machine translation, quality estimation, terminology extraction from comparable texts, the use of speech recognition in the translation process, and human computer interaction and interface design for the professional translation environment. For each of these topics, we describe the experiments we performed and the conclusions drawn, providing an overview of the highlights of the entire SCATE project

Towards a cyber physical system for personalised and automatic OSA treatment

Obstructive sleep apnea (OSA) is a breathing disorder that takes place in the course of the sleep and is produced by a complete or a partial obstruction of the upper airway that manifests itself as frequent breathing stops and starts during the sleep. The real-time evaluation of whether or not a patient is undergoing OSA episode is a very important task in medicine in many scenarios, as for example for making instantaneous pressure adjustments that should take place when Automatic Positive Airway Pressure (APAP) devices are used during the treatment of OSA. In this paper the design of a possible Cyber Physical System (CPS) suited to real-time monitoring of OSA is described, and its software architecture and possible hardware sensing components are detailed. It should be emphasized here that this paper does not deal with a full CPS, rather with a software part of it under a set of assumptions on the environment. The paper also reports some preliminary experiments about the cognitive and learning capabilities of the designed CPS involving its use on a publicly available sleep apnea database

Intelligent Tutoring System for Teaching "Introduction to Computer Science" in Al-Azhar University, Gaza

ITS (Intelligent Tutoring System) is a computer software that supplies direct and adaptive training or response to students without, or with little human teacher interfering. The main target of ITS is smoothing the learning-teaching process using the ultimate technology in computer science. The proposed system will be implemented using the “ITSB” Authoring tool. The book "Introduction To Computer Science" is taught in Al-Azhar University in Gaza as a compulsory subject for students who study at humanities faculties. In this thesis, the researcher demonstrates an intelligent tutoring system for teaching the above mentioned subject. The system was assessed by a group of teachers and students and the results were promising