PromptInfuser: How Tightly Coupling AI and UI Design Impacts Designers' Workflows

Prototyping AI applications is notoriously difficult. While large language model (LLM) prompting has dramatically lowered the barriers to AI prototyping, designers are still prototyping AI functionality and UI separately. We investigate how coupling prompt and UI design affects designers' workflows. Grounding this research, we developed PromptInfuser, a Figma plugin that enables users to create semi-functional mockups, by connecting UI elements to the inputs and outputs of prompts. In a study with 14 designers, we compare PromptInfuser to designers' current AI-prototyping workflow. PromptInfuser was perceived to be significantly more useful for communicating product ideas, more capable of producing prototypes that realistically represent the envisioned artifact, more efficient for prototyping, and more helpful for anticipating UI issues and technical constraints. PromptInfuser encouraged iteration over prompt and UI together, which helped designers identify UI and prompt incompatibilities and reflect upon their total solution. Together, these findings inform future systems for prototyping AI applications

Designing Exploratory Search Systems that Stimulate Memory and Reduce Cognitive Load

From music fans finding new songs in a genre, graphic designers brainstorming ways to depict a message, and journalists scrutinizing documents for angles, people often conduct exploratory searches to understand complex topics. In contrast to traditional search, which is done to quickly answer simple questions, exploratory search is an iterative learning process that involves understanding an information space in order to find useful pieces of information. Exploratory search is composed of two, closely-related sub-processes: (1) information foraging, choosing sources and collecting information, and (2) sensemaking, organizing this information into a mental framework. Both of these sub-processes are cognitively taxing and heavily rely on our memory. For information foraging, users need to read long, complex resources and recognize useful pieces of information. For sensemaking, as users encounter more information, it becomes harder to relate new information to their current knowledge. The spreading activation theory of memory purports that the information we encounter materializes in our working memory, which spreads activation into our long-term memory, enabling us to recall related semantic information to make sense of newly found information. From this theory, this thesis introduces three strategies for creating organizations that better stimulate memory: (1) constructing overviews that are association networks that mimic our memory's structure, (2) incorporating our prior knowledge in these overviews, and (3) providing concrete information to help us make sense of abstract ideas. This thesis demonstrates how to employ these strategies through three exploratory search systems across three domains: (A) SymbolFinder helps graphic designers explore visual symbols for abstract concepts, (B) TastePaths helps music fans explore artists within a genre, and (C) AngleKindling supports journalists explore story angles for a press release. Through this body of work, I demonstrate that by designing exploratory search systems to stimulate our memory, we can make acquiring and making sense of knowledge less cognitively demanding

ConstitutionMaker: Interactively Critiquing Large Language Models by Converting Feedback into Principles

Large language model (LLM) prompting is a promising new approach for users to create and customize their own chatbots. However, current methods for steering a chatbot's outputs, such as prompt engineering and fine-tuning, do not support users in converting their natural feedback on the model's outputs to changes in the prompt or model. In this work, we explore how to enable users to interactively refine model outputs through their feedback, by helping them convert their feedback into a set of principles (i.e. a constitution) that dictate the model's behavior. From a formative study, we (1) found that users needed support converting their feedback into principles for the chatbot and (2) classified the different principle types desired by users. Inspired by these findings, we developed ConstitutionMaker, an interactive tool for converting user feedback into principles, to steer LLM-based chatbots. With ConstitutionMaker, users can provide either positive or negative feedback in natural language, select auto-generated feedback, or rewrite the chatbot's response; each mode of feedback automatically generates a principle that is inserted into the chatbot's prompt. In a user study with 14 participants, we compare ConstitutionMaker to an ablated version, where users write their own principles. With ConstitutionMaker, participants felt that their principles could better guide the chatbot, that they could more easily convert their feedback into principles, and that they could write principles more efficiently, with less mental demand. ConstitutionMaker helped users identify ways to improve the chatbot, formulate their intuitive responses to the model into feedback, and convert this feedback into specific and clear principles. Together, these findings inform future tools that support the interactive critiquing of LLM outputs