How to reach a usable DSL? Moving toward a Systematic Evaluation

Domain-Specific Languages (DSLs) are claimed to increase productivity,while reducing the required maintenance and programming expertise.  In thiscontext, DSL usability by domain experts is a key factor for its successfuladoption.Evidence that support those improvement claims is mostly anecdotal.  Oursystematic literature review showed that a usability evaluation was oftenskipped, relaxed, or at least omitted from papers reporting the development of DSLs.  Thefew exceptions mostly take place at the end of the development process where fixing problems identified is too expensive.We argue that a systematic approach based on User Interface experimentalvalidation  techniques should be used to assess the impact of the new DSLs.  The rationale is that assessing important and specially tailored usability attributes for DSLs early in language construction will ultimately foster a higher productivity of the DSL users. This paper, besides discussing thequality criteria, proposes a development and evaluation process that can be usedto achieve usable DSLs in a better way

A Rule-based Engine to support a Framework for the Experimental Validation of Domain Specific Languages

Software systems are widely used in people daily routines and responsibilities, therefore, systems need to be developed rapidly and efficiently. Domain specific languages (DSLs) are languages that are applied to a specific application domain. Since DSLs provide notations and constructs adapted to a particular domain, they offer gains in expressiveness and ease of use when compared with general-purpose languages (GPLs). Therefore, one of the most important steps in the Software Language Engineering is the evaluation of the languages produced, with the end-users, since the risk of building inappropriate languages, that often do not fit the end users, may decrease productivity. Although DSLs evaluation is one of the most important steps in development process, Software Language Engineers tend to relax the experimental validation of their products due to several reasons like costs (time, means, money, the number of people required, etc.) and required know-how associated with it. The lack of systematic approaches and guidelines to evaluate DSLs, and a comprehensive set of tools may explain this shortcoming in the current state of practice. The Usability Driven DSL development with USE-ME (USE-ME) approach, developed in NOVA-LINCS, "promotes the quality in use of DSLs by building a framework that leverages usability as a main concern". The feedback of the pilot studies was that despite the approach was "more or less easy" to understand it was not easy to model, since "there were too many steps to follow" and the framework did not provide a "guided cycle". So, in order to improve the system usability and the quality of the models produced with USE-ME, we developed a new version of the framework with validation rules implemented with Eclipse Validation Language (EVL) that guide, suggest and validate the Software Language Engineer actions throughout the development process. The validation rules were designed in such a way that the tool educates the user about the process, so that the user makes the best decision regarding his DSL evaluation. We performed two experiments, with different goals. The main goals of the first one was to analyse the effect of validation rules on the USE-ME framework, with respect to their impact on the System Usability Scale, and on the Model Correctness of USE-ME models. We analysed the results and we found evidences of improvements on the System Usability Scale, and on the Model Correctness of models, brought by the addition of the rules. The second experiment was conducted with a research team from Ege University, in Turkey. The main goal of this experiment was to perform a guided evaluation on a DSL related with Multi-Agent Systems, SEA-ML. Since the number of participants was low we cannot draw conclusions regarding this experiment. Despite the significant results from the first experiment further evaluation on the new version of the framework is necessary, this time, with more experienced users and with more complex exercises. With this new experiment, we can compare the results and improve the USE-ME framework

Software Language Engineering: Interaction and Usability Modeling of Language Editors

Background: Domain-Specific Languages (DSLs) are programming languages created to a specific domain that a user has pre-conceived. Multi-Agent Systems (MAS) represent a set of systems interacting within an environment, in which many intelligent agents interact with each other. Usability is a property of something that is "capable of being used"and "convenient and practicable for use". Barišic et al. introduced a conceptual framework that supports the iterative development process of DSLs concerning the usability evaluation. Semantic Web Enabled Agent Modeling Language (SEA_ML) is a DSL that supports the modeling and generation of action-based systems for MAS and the Semantic Web. It is defined by 44 visual notations. Objective: Improve SEA_ML’s usability using "The "Physics"of Notations" principles to create a new visual notation for SEA_ML. Method: (1) Participants test the current notation and the new notation on four exercises. For each exercise, a SUS questionnaire is presented. Participants should have better results on the exercises with the new notation. (2) Participants select the notations for SEA_ML. Participants receive a list with figures including the current and the new notation, alongside a set of descriptions for each of the semantic constructs of SEA_ML. Participants should select more icons from the new notation. Results: With the results gathered from each experience it is not clear that the new visual notations are better than the current notations. Limitation: The results from the guidelines were not evaluated broadly. Conclusion: The results for each experiment are not clear that the new notation is better than the current notation. This thesis is part of a scientific and technological co-operation between NOVA LINCS research center at Universidade Nova de Lisboa, Portugal, and Ege University International Computer Institute, Turkey. regarding the project Developing a Framework on Evaluating Domain specific Modeling Languages for Multi-Agent Systems

Inferring multilingual domain-specific word embeddings from large document corpora

The use of distributed vector representations of words in Natural Language Processing has become established. To tailor general-purpose vector spaces to the context under analysis, several domain adaptation techniques have been proposed. They all require sufficiently large document corpora tailored to the target domains. However, in several cross-lingual NLP domains both large enough domain-specific document corpora and pre-trained domain-specific word vectors are hard to find for languages other than English. This paper aims at tackling the aforesaid issue. It proposes a new methodology to automatically infer aligned domain-specific word embeddings for a target language on the basis of the general-purpose and domain-specific models available for a source language (typically, English). The proposed inference method relies on a two-step process, which first automatically identifies domain-specific words and then opportunistically reuses the non-linear space transformations applied to the word vectors of the source language in order to learn how to tailor the vector space of the target language to the domain of interest. The performance of the proposed method was validated via extrinsic evaluation by addressing the established word retrieval task. To this aim, a new benchmark multilingual dataset, derived from Wikipedia, has been released. The results confirmed the effectiveness and usability of the proposed approach

Design of the CAM model and authoring tool

Students benefit from personalised attention; however, often teachers are unable to provide this. An Adaptive Hypermedia (AH) system can offer a richer learning experience in an educational environment, by giving personalised attention to students. On-line courses are becoming increasingly popular by means of Learning Management Systems (LSM). The aim of the GRAPPLE project is to integrate an AH with major LMS, to provide an environment that delivers personalised courses in a LMS interface. However, designing an AH is a much more complex and time-consuming task, than creating a course in a LMS. Several models and systems were developed previously, but the (re)-usability by educational authors of the adaptation remains limited. To simplify adaptive behaviour authoring for an educational author, a visual environment was selected as being most intuitive. This paper describes a reference model for authoring in a visual way and introduces an authoring tool based upon this model

Software languages engineering: experimental evaluation

Dissertação apresentada na Faculdade de Ciências e Tecnologia da Universidade Nova de Lisboa para obtenção do grau de Mestre em Engenharia InformáticaDomain-Specific Languages (DSLs) are programming languages that offer, through appropriate notation and abstraction, still enough an expressive control over a particular problem domain for more restricted use. They are expected to contribute with an enhancement of productivity, reliability, maintainability and portability, when compared with General Purpose Programming Languages (GPLs). However, like in any Software Product without passing by all development stages namely Domain Analysis, Design, Implementation and Evaluation, some of the DSLs’ alleged advantages may be impossible to be achieved with a significant level of satisfaction. This may lead to the production of inadequate or inefficient languages. This dissertation is focused on the Evaluation phase. To characterize DSL community commitment concerning Evaluation, we conducted a systematic review. The review covered publications in the main fora dedicated to DSLs from 2001 to 2008, and allowed to analyse and classify papers with respect to the validation efforts conducted by DSLs’ producers, where have been observed a reduced concern to this matter. Another important outcome that has been identified is the absence of a concrete approach to the evaluation of DSLs, which would allow a sound assessment of the actual improvements brought by the usage of DSLs. Therefore, the main goal of this dissertation concerns the production of a Systematic Evaluation Methodology for DSLs. To achieve this objective, has been carried out the major techniques used in Experimental Software Engineering and Usability Engineering context. The proposed methodology was validated with its use in several case studies, whereupon DSLs evaluation has been made in accordance with this methodology

Software Usability:A Comparison Between Two Tree-Structured Data Transformation Languages

This paper presents the results of a software usability study, involving both subjective and objective evaluation. It compares a popular XML data transformation language (XSLT) and a general purpose rule-based tree manipulation language which addresses some of the XML and XSLT limitations. The benefits of the evaluation study are discussed