End User Development in the IoT: a Semantic Approach

The Internet of Things (IoT) is, nowadays, a well recognized paradigm. In this field, End User Development (EUD) is a promising approach that allows users to program their devices and services. The representation models adopted by contemporary EUD interfaces, however, are often highly technology-dependent, and the interaction between users and the IoT ecosystem is put to a hard test. The goal of my research is to explore new approaches and tools for helping end-users to program their technological devices and services. For this purpose, I proposed EUPont, an ontological model able to represent abstract and technology independent trigger-action rules, that can be adapted to different contextual situations. EUPont has been evaluated in terms of understandability, completeness, and usefulness. Currently, I am using the semantic features of the model in different research projects, e.g., to optimize the layout of EUD interfaces, and to design a recommender system of trigger-action rules. Preliminary results are promising, and confirm the benefit of using the semantic information of EUPont for helping end-users to better deal with the forthcoming IoT world

Enabling End-User Development in Smart Homes: A Machine Learning-Powered Digital Twin for Energy Efficient Management

End-User Development has been proposed over the years to allow end users to control and manage their Internet of Things-based environments, such as smart homes. With End-User Development, end users are able to create trigger-action rules or routines to tailor the behavior of their smart homes. However, the scientific research proposed to date does not encompass methods that evaluate the suitability of user-created routines in terms of energy consumption. This paper proposes using Machine Learning to build a Digital Twin of a smart home that can predict the energy consumption of smart appliances. The Digital Twin will allow end users to simulate possible scenarios related to the creation of routines. Simulations will be used to assess the effects of the activation of appliances involved in the routines under creation and possibly modify them to save energy consumption according to the Digital Twin’s suggestions

Devices, Information, and People: Abstracting the Internet of Things for End-User Personalization

Nowadays, end users can take advantage of end-user development platforms to personalize the Internet of Things. These platforms typically adopt a vendor-centric abstraction, by letting users to customize each of their smart device and/or online service through different trigger-action rules. Despite the popularity of such an approach, several research challenges in this domain are still underexplored. Which "things" would users personalize, and in which contexts? Are there any other effective abstractions besides the vendor-centric one? Would users adopt different abstractions in different contexts? To answer these questions, we report on the results of a 1-week-long diary study during which 24 participants noted down trigger-action rules arising during their daily activities. Results show that users would adopt multiple abstractions by personalizing devices, information, and people-related behaviors where the individual is at the center of the interaction. We found, in particular, that the adopted abstraction may depend on different factors, ranging from the user profile to the context in which the personalization is introduced. While users are inclined to personalize physical objects in the home, for example, they often go "beyond devices" in the city, where they are more interested in the underlying information. Our findings identify new design opportunities in HCI to improve the relationship between the Internet of Things, personalization paradigms, and users

FaceMashup: An end-user development tool for social network data

Every day, each active social network user produces and shares texts, images and videos. While developers can access such data through application programming interfaces (APIs) for creating games, visualizations and routines, end users have less control on such information. Their access is mediated by the social application features, which limits them in combining sources, filtering results and performing actions on groups of elements. In order to fill this gap, we introduce FaceMashup, an end user development (EUD) environment supporting the manipulation of the Facebook graph. We describe the tool interface, documenting the choices we made during the design iterations. Data types are represented through widgets containing user interface (UI) elements similar to those used in the social network application. Widgets can be connected with each other with the drag and drop of their inner fields, and the application updates their content. Finally, we report the results of a user-test on the FaceMashup prototype, which shows a good acceptance of the environment by end-users

My IoT Puzzle: Debugging IF-THEN Rules Through the Jigsaw Metaphor

End users can nowadays define applications in the format of IF-THEN rules to personalize their IoT devices and online services. Along with the possibility to compose such applications, however, comes the need to debug them, e.g., to avoid unpredictable and dangerous behaviors. In this context, different questions are still unexplored: which visual languages are more appropriate for debugging IF-THEN rules? Which information do end users need to understand, identify, and correct errors? To answer these questions, we first conducted a literature analysis by reviewing previous works on end-user debugging, with the aim of extracting design guidelines. Then, we developed My IoT Puzzle, a tool to compose and debug IF-THEN rules based on the Jigsaw metaphor. My IoT Puzzle interactively assists users in the debugging process with different real-time feedback, and it allows the resolution of conflicts by providing textual and graphical explanations. An exploratory study with 6 participants preliminary confirms the effectiveness of our approach, showing that the usage of the Jigsaw metaphor, along with real-time feedback and explanations, helps users understand and fix conflicts among IF-THEN rules

HeyTAP: Bridging the Gaps Between Users' Needs and Technology in IF-THEN Rules via Conversation

In the Internet of Things era, users are willing to personalize the joint behavior of their connected entities, i.e., smart devices and online service, by means of IF-THEN rules. Unfortunately, how to make such a personalization effective and appreciated is still largely unknown. On the one hand, contemporary platforms to compose IF-THEN rules adopt representation models that strongly depend on the exploited technologies, thus making end-user personalization a complex task. On the other hand, the usage of technology-independent rules envisioned by recent studies opens up new questions, and the identification of available connected entities able to execute abstract users' needs become crucial. To this end, we present HeyTAP, a conversational and semantic-powered trigger-action programming platform able to map abstract users' needs to executable IF-THEN rules. By interacting with a conversational agent, the user communicates her personalization intentions and preferences. User's inputs, along with contextual and semantic information related to the available connected entities, are then used to recommend a set of IF-THEN rules that satisfies the user's needs. An exploratory study on 8 end users preliminary confirms the effectiveness and the appreciation of the approach, and shows that HeyTAP can successfully guide users from their needs to specific rules

Empowering End Users in Debugging Trigger-Action Rules

End users can program trigger-action rules to personalize the joint behavior of their smart devices and online services. Trigger-action programming is, however, a complex task for non-programmers and errors made during the composition of rules may lead to unpredictable behaviors and security issues, e.g., a lamp that is continuously fashing or a door that is unexpectedly unlocked. In this paper, we introduce EUDebug, a system that enables end users to debug trigger-action rules. With EUDebug, users compose rules in a web-based application like IFTTT. EUDebug highlights possible problems that the set of all defned rules may generate and allows their step-by-step simulation. Under the hood, a hybrid Semantic Colored Petri Net (SCPN) models, checks, and simulates trigger-action rules and their interactions. An exploratory study on 15 end users shows that EUDebug helps identifying and understanding problems in trigger-action rules, which are not easily discoverable in existing platforms

AwareNotifications: Multi-Device Semantic Notification Handling with User-Defined Preferences

With the increase of connected devices and online services, the number of notifications received by each person is growing. Although notifications are useful to inform users about important information such as new messages and events, the continuous interruptions, the notification duplication, and the rigid delivery can be sources of discomfort. To overcome these issues, we present AwareNotifications, an intelligent system based on user-defined preferences to manage multi-device notifications. AwareNotifications is powered by Semantic Web technologies. By directly exploiting user preferences in the semantic reasoning process, the system is able to identify suitable device(s), modality, and moment(s) to deliver the incoming user notifications. We evaluated AwareNotifications in a user study with 15 participants, in which we compared our system with the "traditional" notification delivery system. The study confirms the perceived effectiveness of AwareNotifications, and provides insights to further improve the system

From Users' Intentions to IF-THEN Rules in the Internet of Things

In the Internet of Things era, users are willing to personalize the joint behavior of their connected entities, i.e., smart devices and online service, by means of trigger-action rules such as "IF the entrance Nest security camera detects a movement, THEN blink the Philips Hue lamp in the kitchen." Unfortunately, the spread of new supported technologies make the number of possible combinations between triggers and actions continuously growing, thus motivating the need of assisting users in discovering new rules and functionality, e.g., through recommendation techniques. To this end, we present HeyTAP², a semantic Conversational Search and Recommendation (CSR) system able to suggest pertinent IF-THEN rules that can be easily deployed in different contexts starting from an abstract user's need. By exploiting a conversational agent, the user can communicate her current personalization intention by specifying a set of functionality at a high level, e.g., to decrease the temperature of a room when she left it. Stemming from this input, HeyTAP² implements a semantic recommendation process that takes into account a) the current user’s intention, b) the connected entities owned by the user, and c) the user's long-term preferences revealed by her profile. If not satisfied with the suggestions, the user can converse with the system to provide further feedback, i.e., a short-term preference, thus allowing HeyTAP² to provide refined recommendations that better align with the her original intention. We evaluate HeyTAP² by running different offline experiments with simulated users and real-world data. First, we test the recommendation process in different configurations, and we show that recommendation accuracy and similarity with target items increase as the interaction between the algorithm and the user proceeds. Then, we compare HeyTAP² with other similar baseline recommender systems. Results are promising and demonstrate the effectiveness of HeyTAP² in recommending IF-THEN rules that satisfy the current personalization intention of the user

Prototyping tool design: Prototyping user experience in systems with multiple devices & sensors

Many recognize the phrase Smart Home, but few have actually experienced it. The world of Internet of Things (IoT) is all over the IT industry. The hardware already exists. So, why do we not see these kinds of solutions in reality? A simple Smart Home experience could be a system that consists of a motion sensor and a coffee machine. The motion sensor is placed above your apartment door. When it senses that you come home from work, the sensor signals your coffee machine to start brewing a cup of coffee. The above example is easy enough for most users to imagine and design, but for them to actually create and test it is almost impossible. The skill of programming sensors to communicate with objects is still highly technical. This thesis addresses this problem and describes solutions for users with no technical background to more easily create this type of experiences. The final prototype design of this project is a software tool design that lets users with no experience in programming create simple User Experiences that includes various sensors and multiple devices. In the prototype, these experiences can also be manually simulated in the tool without using physical sensors connected to the software