A considerable amount of research has been carried out towards enabling average users to customize their smart homes through trigger-action ("if... then...") programming. However, inhabitants of such smart environments keep having problems understanding, administering, troubleshooting, and deriving benefits from the technologies employed in their homes. By synthesizing a broad body of research on end-user programming in smart homes with observations of commercial products
and our own experiences, we provide a set of guidelines for designers of future interfaces and tools. Stemming from them, we present the design and the initial evaluation of HomeRules, a mobile and tangible application for end-user programming in smart homes

Corno, Fulvio

DE RUSSIS, Luigi

English

DE RUSSIS, LUIGI

CORNO, Fulvio

PORTO@iris (Publications Open Repository TOrino - Politecnico di Torino)

04 August 2020POLITECNICO DI TORINORepository ISTITUZIONALEHomeRules: A Tangible End-User Programming Interface for Smart Homes / DE RUSSIS, Luigi; Corno, Fulvio. - (2015),pp. 2109-2114. ((Intervento presentato al convegno CHI'15: CHI Conference on Human Factors in Computing Systemstenutosi a Seoul, Republic of Korea nel April 18 - 23, 2015.OriginalHomeRules: A Tangible End-User Programming Interface for Smart HomesPublisher:PublishedDOI:10.1145/2702613.2732795Terms of use:openAccessPublisher copyright(Article begins on next page)This article is made available under terms and conditions as specified in the  corresponding bibliographic description inthe repositoryAvailability:This version is available at: 11583/2588215 since:ACMHomeRules: A Tangible End-UserProgramming Interface for SmartHomesLuigi De RussisPolitecnico di TorinoCorso Duca degli Abruzzi, 54Torino, 10129 Italyluigi.derussis@polito.itFulvio CornoPolitecnico di TorinoCorso Duca degli Abruzzi, 54Torino, 10129 Italyfulvio.corno@polito.itPermission to make digital or hard copies of part or all of this work for personalor classroom use is granted without fee provided that copies are not made ordistributed for profit or commercial advantage and that copies bear this noticeand the full citation on the first page. Copyrights for third-party componentsof this work must be honored. For all other uses, contact the Owner/Author.Copyright is held by the owner/author(s).CHI’15 Extended Abstracts, Apr 18-23, 2015, Seoul, Republic of KoreaACM 978-1-4503-3146-3/15/04.http://dx.doi.org/10.1145/2702613.2732795AbstractA considerable amount of research has been carried outtowards enabling average users to customize their smarthomes through trigger-action (“if. . . then. . . ”)programming. However, inhabitants of such smartenvironments keep having problems understanding,administering, troubleshooting, and deriving benefits fromthe technologies employed in their homes. By synthesizinga broad body of research on end-user programming insmart homes with observations of commercial productsand our own experiences, we provide a set of guidelinesfor designers of future interfaces and tools. Stemmingfrom them, we present the design and the initialevaluation of HomeRules, a mobile and tangibleapplication for end-user programming in smart homes.Author KeywordsSmart home; home automation; end-user programming;mobile programming; guidelinesACM Classification KeywordsH.5.m [Information interfaces and presentation (e.g.,HCI)]: Miscellaneous.IntroductionIn the last years, smart homes gained a new momentum,thanks to an increased availability of commercial solutionsand to steadily reducing costs. As a result, a new body ofknowledge is now available to the research community andto the industry, and further explorations “in the wild” arefinally possible.Mennicken et al., reporting the results from a qualitativestudy on various stakeholders of smart hometechnologies [4], discovered that home inhabitants,particularly those who do not have any technicalknowledge, have problems understanding, administeringand troubleshooting smart home technologies. This meansthat a true collaboration between the home and theirinhabitants is still unreached: in fact, autonomoustechnologies often leave users feeling out of control,without the possibility to adjust the level of autonomy oftheir home according to their needs [1]. To avoid thisfeeling, researchers and practitioners are currentlyproposing interfaces and tools that allow homeinhabitants to explicitly delegate specific tasks, maybeboring or repetitive, to their smart homes.A promising approach to delegate such tasks and createconnections between individual devices seems to be toexpress the inhabitants needs, wishes and desiderata in amore natural, less technical form [5], trying to avoid usingthe vocabulary and the typical metaphors adopted by andfor people with technical background. In this way, aninhabitant-centered approach is ensured: users are alwaysin the loop and perceive the smart home as somethingboth convenient and useful. Moreover, according toNewman [6], end-user configurability is a key factor insmart home applications.Building upon these insights, we provide a set ofguidelines that emerge from a literature review and froman analysis of current interfaces and tools for end-userprogramming in smart homes. Starting from theseguidelines, we present the design and the initial evaluationof HomeRules, a mobile and tangible application toempower home inhabitants with few or no technical skillsto effectively “program” their smart home.Literature ReviewSeveral interfaces and tools for end-user programming insmart homes are present in the literature. Suchapplications, typically, let users define some rules1 forprogramming, without writing any code, the environmentwhere they live or operate.To extrapolate a set of guidelines for designers of futureend-users programming tools for smart homes, we firstassembled a list of works known to the authors. We thencompleted the list by adding any of the search results onACM and IEEE Xplore Digital Libraries for the keywords“end-user programming”, “trigger-action programming”,“activity delegation” and “context-aware computing”combined with “smart home” or “home automation”, andnot already included in the known body of works. Thefinal list comprises 47 papers, counted after cleaning theresults from unrelated works and excluding patents,keynote talks, front covers and similar content. In thispaper, for brevity, we report a few of the most relevantworks, only.Among the analyzed papers, the work of Dey et al. [2] is,probably, the most cited and can be considered as a“cornerstone” for several subsequent papers. In theirwork, they present iCAP, a visual, PC-based andrule-based system for building context-aware applicationsfor smart environments that does not require users towrite any code. iCAP supports trigger-action rules (“if1also called “context-aware applications” or “trigger-action pro-grams”something happens, then do something else”) with “and”and “or” conditional operators. The trigger-action formatis one of the most used in the analyzed papers and can beuseful and usable for end-user programming in smarthomes, according to Ur et al. [7]. Trigger-action issometimes extended to trigger-constraint-action, alsoknown as Event-Condition-Action (ECA).Lee et al., in [3], propose GALLAG Strip, a mobile andtangible tool for creating context-aware applications.Various insights emerge from the in-depth validation ofGALLAG Strip, like the suggestion of mixing tangible andnon-tangible approaches for augmenting the overall userexperience or the limitation of using the “small” screen ofa smartphone. Other interesting suggestions are reportedin the paper of Ur et al. [7], where the authors investigatethe practicality of end-user programming to customizesmart home devices, evaluating thousands oftrigger-action programs shared on the website IFTTT (“IfThis Then That”)2 and conducting a usability test withmore than 200 participants.GuidelinesThe guidelines proposed in this section synthesize thefindings that emerge from the literature review and aim atproviding a clear guidance for designers and engineersinterested in the creation of end-user programming toolsfor a smart home. They avoid giving any insight aboutrule clashing, conflicts or debugging. In writing suchguidelines, we also consider our own previous experiencesand some well-known commercial products that enableaverage users to engage in trigger-action programmingwith household devices, like IFTTT or the WigWagsystem3. The extracted guidelines are:2http://ifttt.com, last visited on December 29, 20143http://www.wigwag.com, last visited on December 29, 20141. Use the ECA format for representing smart homerules, in a visual way. Trigger-action is one of themost common formats in the reviewed papers, agood match with users mental model [2] and can beuseful and usable for end-user programming insmart homes [7].2. Avoid using natural language for rules creation.Typically used words for specifying events andconditions (i.e., “if” and “when”) are semanticallyclose in many languages. This may confuse theusers during the rule creation process.3. Provide a simple and clear visualization of existingrules. With a simple and clear visualization of rules,home inhabitants accept and use better their smarthome, especially those who have a lack of interest inparticipating actively in the configuration of smarthome technologies.4. Provide both tangible and non-tangible interaction.A mixed tangible and non-tangible approach seemsappropriate to encompass users with different needsand programming skills [3].5. Provide a step-by-step creation mechanism forfirst-time users. Novice users can largely benefitfrom a step-by-step rule creation mechanism, tobetter understand the trigger-action model and themeaning of “creating a smart home rule”.6. Handle time-related properties separately. Temporalrelationships inside rules (e.g., “at 6 o’ clock”, “for2 minutes”, etc.) and chronological operationsshould be treated in a dedicated, specific way toisolate their complexity.7. Apply a mobile approach, targeting screens largerthan 6”. Small screens can be a issue during rulecreation [3].8. Adopt single touch (or click) interactions instead ofmore complex operations. As reported in thefollowing section, users seem to find “easier” theusage of single touch (or click) for rule compositioninstead of more complex operations, likedrag-and-drop.9. Supplement the visual representation with otherfeedback mechanisms, like sound. Particularly usefulfor tangible programming, sound or other type offeedback can signal the success of an operationwithout requiring the user to look at the screen.10. Prefer testing “in the wild”. Several issues that canemerge in realizing a rule composition tool aredifficult to catch and understand in a controlledsetting. Evaluation should, therefore, be performedby including also sessions in the field, given thegreat availability of commercial smart home devices.Initial PrototypesFigure 1: The “single touch”paper prototype.Figure 2: The “drag-and-drop”paper prototype.With the previous guidelines in mind, we designed twopaper prototypes with the goal of creating a tablet-based,end-user and tangible programming interface for a smarthome.Both prototypes express a rule in term of events,conditions and actions (according to the ECA model); inparticular, each rule is triggered by an event only (or aseries of mutually disjoint events), have to optionallysatisfy a set of conditions, and may execute one or moreactions. The prototypes present three areas: a DevicesArea that collects all smart home devices; a CompositionArea, used for the actual rule composition; and a ChoiceArea that shows the events, conditions or actions availablefor the selected device. The two prototypes mainly differfor the modality in which a rule can be composed: theformer (Figure 1) uses single touch for rule composition,while the latter (Figure 2) employs drag-and-drop for thesame operation.Three HCI/Ubicomp experts evaluated the prototypes toidentify possible weaknesses in the concepts and thegeneral interface design. After a minor revision, thoseprototypes were evaluated using a think-aloud,within-subject study with two groups of 6 participants (3male, 3 female) each: one group composed by people withlittle to no technical experience and no programmingexperience, and the other composed by participants with acomputer science or engineering background. Participantsages ranged between 24 and 30, with a mean age of 26.2years (SD = 2.6) for the “non-technical” group and of25.7 years (SD = 2.3) for the other. All of them livedwith at least one more person and without their parents.Our main goal in this initial evaluation was to learn aboutwhether participants understood the presented ECAformat, which interaction modality they preferred, andwhether they envisioned using them.Participants in both groups composed successfully at leastone rule with each prototype and indicated that theyunderstood the used format and formalism. Both groupsstrongly preferred the interaction modality of the firstprototype (i.e., single touch) as they found it “simpler”and “less demanding [from a cognitive standpoint]”. Twonon-technical participants had some minor difficulties inunderstanding the difference between an event and acondition: they suggested to better highlight thisdifference, maybe using different icons for distinguishingthe events that trigger the rule from the conditions. Whenasked about whether they will use a tangible approach forcomposing such rules, half of the participants (3 for eachgroup) found it useful but not essential, thus confirmingthe fourth guideline. Because our participants expressed astrong interest in the first prototype, we decided to focusour subsequent efforts on it, developing an interactiveprototype named HomeRules.Interactive Prototype: HomeRulesTo better investigate the applicability and the consistencyof the reported guidelines, we revised the HomeRulesdesign, addressing issues identified in the paper prototypeevaluation and adding missing functions. The paperprototypes evaluated by the end-users, in fact, coverguidelines 1, 2, 3, 4, 7, and 8, partially or completely. Theinteractive prototype completes the tangible part(guideline 4), adds the step-by-step rule compositionmechanism (guideline 5) and handles temporalrelationship (guideline 6). It also adds the various menusand feedback mechanisms (guideline 9). All thesefunctions were evaluated by the three experts.Figure 3: The “InteractiveLearning” modality at work.Figure 4: A possible suggestionas shown in HomeRules.In particular, the tangible interaction is enabled from therule composition screen by choosing the “InteractiveLearning” option. This modality puts the application in alistening mode: HomeRules waits until the user performsany operation in the home and then reports it. In case ofambiguity, the application lets the user choose whetherthe perceived operation is an event, a condition or anaction (Figure 3). The step-by-step rule composition isrealized in two modalities: a “traditional” guided tutorialand a “suggestion” modality that shows the nextoperation(s) to be performed to successfully compose arule (like in Figure 4). Up to now, temporal relationshipsare handled through a “clock” and a “calendar” virtualdevices, which can be used as events or conditions.The current prototype of HomeRules is an Android 4.x app(see Figure 5), optimized for a 10” screen. HomeRulestakes the needed data from an instance of Dog4, anopen-source smart home gateway, that runs in one of ourUniversity offices. HomeRules uses the Dog WebSocketAPI for getting the list of the smart home devices, theirproperties, and for interacting with the devices in thetangible modality. In the current state, all the rulescreated with HomeRules are saved in JSON format andsent to the Dog gateway, that is responsible for theirexecution. In the same way, at start time, HomeRulesasks Dog for already existing rules to allow their editing.Figure 5: Screenshot of the current version of HomeRules.4http://dog-gateway.github.io, last visited on December29, 2014Conclusions and Future WorksThe contribution of this work in progress is twofold. Fromone side, it proposes a set of guidelines, extrapolated fromthe literature, that should constitute a strong foundationand starting point for designers and engineers of futureend-user programming tools for smart homes. From theother side, it presents HomeRules, a prototype interfacethat builds upon the reported guidelines and seems toconfirm their validity; moreover, its preliminary evaluationbetter clarifies and details most of the guidelines.Currently, we are preparing a user evaluation of theinteractive prototype of HomeRules. As a future work, weplan to extend HomeRules to encompass issues likevisualization and explanation of rule conflicts, debuggingand preview of created rules, and high-level handling andconcatenation of multiple rules. Finally, we aim atdeveloping a prototype suitable for real-world deployment,to better understand the effective usage of the applicationand obtain further information on how end-userprogramming can improve the smart home experience.AcknowledgementsThe authors wish to thank the MSc student RosalbaCastro for developing the interactive prototype and all theanonymous participants of our study.References[1] Ball, M., and Callaghan, V. Managing control,convenience and autonomy - A study of agentautonomy in intelligent environments. In Agents andAmbient Intelligence - Achievements and Challenges inthe Intersection of Agent Technology and AmbientIntelligence. 2012, 159–196.[2] Dey, A. K., Sohn, T., Streng, S., and Kodama, J.iCAP: Interactive Prototyping of Context-awareApplications. In Proceedings of the 4th InternationalConference on Pervasive Computing, PERVASIVE’06,Springer-Verlag (Berlin, Heidelberg, 2006), 254–271.[3] Lee, J., Gardun˜o, L., Walker, E., and Burleson, W. ATangible Programming Tool for Creation ofContext-aware Applications. In Proceedings of the2013 ACM International Joint Conference onPervasive and Ubiquitous Computing, UbiComp ’13,ACM (New York, NY, USA, 2013), 391–400.[4] Mennicken, S., and Huang, E. M. Hacking theNatural Habitat: An In-the-wild Study of SmartHomes, Their Development, and the People Who Livein Them. In Proceedings of the 10th InternationalConference on Pervasive Computing, Pervasive’12,Springer-Verlag (Berlin, Heidelberg, 2012), 143–160.[5] Mennicken, S., Vermeulen, J., and Huang, E. M.From Today’s Augmented Houses to Tomorrow’sSmart Homes: New Directions for Home AutomationResearch. In Proceedings of the 2014 ACMInternational Joint Conference on Pervasive andUbiquitous Computing, UbiComp ’14, ACM (NewYork, NY, USA, 2014), 105–115.[6] Newman, M. W. Now we’re cooking: Recipes forend-user service composition in the digital home. InCHI 2006 Workshop IT@Home (2006).[7] Ur, B., McManus, E., Pak Yong Ho, M., and Littman,M. L. Practical Trigger-action Programming in theSmart Home. In Proceedings of the SIGCHIConference on Human Factors in Computing Systems,CHI ’14, ACM (New York, NY, USA, 2014), 803–812.

HomeRules: A Tangible End-User Programming Interface for Smart Homes

https://iris.polito.it/retrieve/handle/11583/2588215/64245/wip365-de-russisA.pdf

HomeRules: A Tangible End-User Programming Interface for Smart Homes

Abstract

Similar works

Full text

Available Versions

PORTO@iris (Publications Open Repository TOrino - Politecnico di Torino)