PromptAid: Prompt Exploration, Perturbation, Testing and Iteration using Visual Analytics for Large Language Models

Large Language Models (LLMs) have gained widespread popularity due to their ability to perform ad-hoc Natural Language Processing (NLP) tasks with a simple natural language prompt. Part of the appeal for LLMs is their approachability to the general public, including individuals with no prior technical experience in NLP techniques. However, natural language prompts can vary significantly in terms of their linguistic structure, context, and other semantics. Modifying one or more of these aspects can result in significant differences in task performance. Non-expert users may find it challenging to identify the changes needed to improve a prompt, especially when they lack domain-specific knowledge and lack appropriate feedback. To address this challenge, we present PromptAid, a visual analytics system designed to interactively create, refine, and test prompts through exploration, perturbation, testing, and iteration. PromptAid uses multiple, coordinated visualizations which allow users to improve prompts by using the three strategies: keyword perturbations, paraphrasing perturbations, and obtaining the best set of in-context few-shot examples. PromptAid was designed through an iterative prototyping process involving NLP experts and was evaluated through quantitative and qualitative assessments for LLMs. Our findings indicate that PromptAid helps users to iterate over prompt template alterations with less cognitive overhead, generate diverse prompts with help of recommendations, and analyze the performance of the generated prompts while surpassing existing state-of-the-art prompting interfaces in performance

Formal representation of ambulatory assessment protocols in HTML5 for human readability and computer execution

Ambulatory assessment (AA) is a research method that aims to collect longitudinal biopsychosocial data in groups of individuals. AA studies are commonly conducted via mobile devices such as smartphones. Researchers tend to communicate their AA protocols to the community in natural language by describing step-by-step procedures operating on a set of materials. However, natural language requires effort to transcribe onto and from the software systems used for data collection, and may be ambiguous, thereby making it harder to reproduce a study. Though AA protocols may also be written as code in a programming language, most programming languages are not easily read by most researchers. Thus, the quality of scientific discourse on AA stands to gain from protocol descriptions that are easy to read, yet remain formal and readily executable by computers. This paper makes the case for using the HyperText Markup Language (HTML) to achieve this. While HTML can suitably describe AA materials, it cannot describe AA procedures. To resolve this, and taking away lessons from previous efforts with protocol implementations in a system called TEMPEST, we offer a set of custom HTML5 elements that help treat HTML documents as executable programs that can both render AA materials, and effect AA procedures on computational platforms.</p

STUDIES IN THE EVALUATION OF A DOMAIN-INDEPENDENT NATURAL LANGUAGE QUERY SYSTEM

There is growing consensus that some of the most crucial questions concerning the feasibility and desirability of natural language interfaces to databases can only be resolved by empirical research. This paper reports the results of several empirical studies which investigated the same domain-independent natural language query system, using various applications in two different natural languages - English and German. Taken together, these experiments involved about 100 subjects and over 12,000 queries, constituting the bulk of empirical evaluations of natural query language systems reported to date. Some definitive results are derived from the combined experience, and plans are outlined to resolve several of the remaining issues.Information Systems Working Papers Serie

Ask Your Data - Supporting Data Science Processes by Combining AutoML and Conversational Interfaces

Data Science is increasingly applied for solving real-life problems, both in industry and in academic research, but mastering Data Science requires an interdisciplinary education that is still scarce on the market. Thus, there is a growing need for user-friendly tools that allow domain experts to directly apply data analysis methods to their datasets, without involving a Data Science expert. In this scenario, we present DSBot, an assistant that can analyze the user data and produce answers by mastering several Data Science techniques. DSBot understands the research question with the help of conversation interaction, produces a data science pipeline and automatically executes the pipeline in order to generate analysis. The strength of DSBot lies in the design of a rich domain specific language for modeling data analysis pipelines, the use of a suitable neural network for machine translation of research questions, the availability of a vast dictionary of pipelines for matching the translation output, and the use of natural language technology provided by a conversational agent. We benchmarked DSBot on two sets of 100 natural language questions and of 30 prediction tasks. We empirically evaluated the translation capabilities and the autoML performance of the system. In the translation task, it obtains a median BLEU score of 0.75. In prediction tasks, DSBot outperforms TPOT, an autoML tool, in 19 datasets out of 30

Categorizing Non-Functional Requirements Using a Hierarchy in UML.

Non-functional requirements (NFRs) are a subset of requirements, the means by which software system developers and clients communicate about the functionality of the system to be built. This paper has three main parts: first, an overview of how non-functional requirements relate to software engineering is given, along with a survey of NFRs in the software engineering literature. Second, a collection of 161 NFRs is diagrammed using the Unified Modelling Language, forming a tool with which developers may more easily identify and write additional NFRs. Third, a lesson plan is presented, a learning module intended for an undergraduate software engineering curriculum. The results of presenting this learning module to a class in Spring, 2003 is presented

Deconstructing NLG Evaluation: Evaluation Practices, Assumptions, and Their Implications

There are many ways to express similar things in text, which makes evaluating natural language generation (NLG) systems difficult. Compounding this difficulty is the need to assess varying quality criteria depending on the deployment setting. While the landscape of NLG evaluation has been well-mapped, practitioners' goals, assumptions, and constraints -- which inform decisions about what, when, and how to evaluate -- are often partially or implicitly stated, or not stated at all. Combining a formative semi-structured interview study of NLG practitioners (N=18) with a survey study of a broader sample of practitioners (N=61), we surface goals, community practices, assumptions, and constraints that shape NLG evaluations, examining their implications and how they embody ethical considerations.Comment: Camera Ready for NAACL 2022 (Main Conference

The impact of artificial intelligence on business: benefits and ethical challenges on customer level

Artificial intelligence is nowadays transforming the industries all around the world. Many businesses are confused whether investing in that new technology and taking a part of this race or taking the risk of losing a competitive advantage in the market. According to data, artificial intelligence AI will lead to an estimated $15.7 trillion, which is 26% increase in global GDP by 2030. This study shows the impact of AI on businesses especially the benefits and ethical challenges by analyzing data collected from a sample of more than 100 random people. Keywords: Artificial Intelligence, business, consumer behavior, marketing strategy, AI investment, ethics DOI: 10.7176/JMCR/81-06 Publication date:August 31st 202

Structuring visual exploratory analysis of skill demand

The analysis of increasingly large and diverse data for meaningful interpretation and question answering is handicapped by human cognitive limitations. Consequently, semi-automatic abstraction of complex data within structured information spaces becomes increasingly important, if its knowledge content is to support intuitive, exploratory discovery. Exploration of skill demand is an area where regularly updated, multi-dimensional data may be exploited to assess capability within the workforce to manage the demands of the modern, technology- and data-driven economy. The knowledge derived may be employed by skilled practitioners in defining career pathways, to identify where, when and how to update their skillsets in line with advancing technology and changing work demands. This same knowledge may also be used to identify the combination of skills essential in recruiting for new roles. To address the challenges inherent in exploring the complex, heterogeneous, dynamic data that feeds into such applications, we investigate the use of an ontology to guide structuring of the information space, to allow individuals and institutions to interactively explore and interpret the dynamic skill demand landscape for their specific needs. As a test case we consider the relatively new and highly dynamic field of Data Science, where insightful, exploratory data analysis and knowledge discovery are critical. We employ context-driven and task-centred scenarios to explore our research questions and guide iterative design, development and formative evaluation of our ontology-driven, visual exploratory discovery and analysis approach, to measure where it adds value to users’ analytical activity. Our findings reinforce the potential in our approach, and point us to future paths to build on

Information Outlook, January 1997

Volume 1, Issue 1https://scholarworks.sjsu.edu/sla_io_1997/1000/thumbnail.jp