Identifikation problematischer Substantivierungen in natürlichsprachigen Anforderungsdokumenten

Nominalisierungen in natürlichsprachigen Anforderungsdokumenten können potentiell gefährlich für das Gelingen des Projekts sein. Bei dem Einsatz von Nominalisierungen wird häufig zusätzliche Information weggelassen, die aber für das vollständige Verständnis benötigt wird. So ist etwa bei der Nominalisierung Anmeldung ohne eine nähere Spezifikation unklar, wo sich wer auf welche Weise anmeldet. Daher ist eine Erkennung von unterspezifizierten Nominalisierungen wichtig, damit eine Fehlinterpretation der Anforderung, was eine potentielle Gefahr für das Scheitern des Projekts darstellt, vermieden wird. Die Fachliteratur rät davon ab Nominalisierungen zu verwenden. Bei der bisherigen Erkennung von Nominalisierungen mit dem Werkzeug RESI wurden ebenso alle Nominalisierungen aufgezeigt. Allerdings ist nicht jede Nominalisierung an sich kritisch. Somit erhielt der Nutzer eine unnötig hohe Anzahl an Warnungen, was die Akzeptanz des Werkzeuges schmälert. In dieser Arbeit wird eine Kategorisierung von Nominalisierungen vorgestellt, die sie in potentiell gefährliche und ungefährliche Nominalisierungen einteilt. Mit dieser Kategorisierung wird eine Möglichkeit gezeigt, wie unproblematische Nominalisierungen erkannt werden können. Dafür wird mit Hilfe von Ontologien und einem Parser die Nominalisierung und der Satz, in dem sie steht, auf spezifizierende Wörter, Nominalphrasen etc. untersucht. Mit diesem Ansatz wurden Lastenhefte der Daimler AG untersucht. Das Prüfungskorpus beinhaltete knapp 60.000 Wörter, welches über 1.100 Nominalisierungen beinhaltet. Davon konnte ein Großteil als unproblematisch gefiltert und damit 129 als problematisch identifiziert werden. Durch diesen Ansatz konnte somit eine große Anzahl von Fehlwarnungen unterdrückt werden, wodurch die Akzeptanz beim Nutzer und die Praxistauglichkeit gesteigert wird

Themenextraktion zur Domänenauswahl für Programmierung in natürlicher Sprache

Für den Menschen sind Kontexte von Anweisungen für die Programmierung in natürlicher Sprache einfach ersichtlich, für den Rechner ist dies nicht der Fall. Eine Art des Kontextwissens ist das Verständnis der Themen. Hierfür wird im Rahmen des PARSE-Projekts zur Programmierung in natürlicher Sprache ein Ansatz zur Themenextraktion vorgestellt, welcher mit Hilfe der Wikipedia als Informationsquelle passende Themen aus den Anweisungen extrahiert. Dafür wird eine Auflösung von mehrdeutigen Nomen benötigt, weshalb in dieser Arbeit ebenfalls ein Werkzeug dafür entwickelt wird. Als einen konkreten Anwendungsfall für die extrahierten Themen wird die Auswahl von passenden Ontologien zu diesen Themen angegangen. Durch diese Auswahl wird ermöglicht, statt einer großen allgemeinen Ontologie mehrere kleine domänenspezifische Ontologien einzusetzen. Auf einem Korpus mit Anweisungen für die Programmierung in natürlicher Sprache konnte für die Auflösung von Mehrdeutigkeiten eine Genauigkeit von 87,80% erreicht werden. Um die Themenextraktion zu evaluieren, wurde eine Umfrage durchgeführt, die ergab, dass das erste extrahierte Thema in bis zu 63,6% der Fälle treffend war. Unter den ersten vier extrahierten Themen konnte sogar in knapp 91% der Fälle mindestens ein passendes Thema gefunden werden. Zuletzt wurden in der Evaluation der Ontologieauswahl gute Ergebnisse erzielt, wobei ein F1-Maß von 90,67% und ein F2-Maß von 89,94% erreicht wurden. Dabei wurde festgestellt, dass Schwierigkeiten auftreten, wenn bei der Auswahl mehrere Ontologien benötigt werden. Der Ansatz könnte zukünftig daher mit einer verbesserten Erkennung von unterschiedlichen Themen in der Eingabe weiter ausgebaut werden

Knowledge-based Sense Disambiguation of Multiword Expressions in Requirements Documents

Understanding the meaning and the senses of expressions is essential to analyze natural language requirements. Disambiguation of expressions in their context is needed to prevent misinterpretations. Current knowledge-based disambiguation approaches only focus on senses of single words and miss out on linking the shared meaning of expressions consisting of multiple words. As these expressions are common in requirements, we propose a sense disambiguation approach that is able to detect and disambiguate multiword expressions. We use a two-tiered approach to be able to use different techniques for detection and disambiguation. Initially, a conditional random field detects multiword expressions. Afterwards, the approach disambiguates these expressions and retrieves the corresponding senses using a knowledge-based approach. The knowledge-based approach has the benefit that only the knowledge base has to be exchanged to adapt the approach to new domains and knowledge. Our approach is able to detect multiword expressions with an $\text{F}_{1}$-score of 88.4% in an evaluation on 997 requirement sentences. The sense disambiguation achieves up to 57% $\text{F}_{1}$-score

A Taxonomy for Design Decisions in Software Architecture Documentation

A software system is the result of all design decisions that were made during development and maintenance. Documentation, such as software architecture documentation, captures a variety of different design decisions. Classifying the kinds of design decisions facilitates various downstream tasks by enabling more targeted analyses. In this paper, we propose a taxonomy for design decisions in software architecture documentation to primarily support consistency checking. Existing taxonomies about design decisions have different purposes and do not fit well because they are too coarse. We take an iterative approach, starting with an initial taxonomy based on literature and considerations regarding consistency checking. Then, we mine open-source repositories to extract 17 software architecture documentations that we use to refine the taxonomy. We evaluate the resulting taxonomy with regard to purpose, structure, and application. Additionally, we explore the automatic identification and classification of design decisions in software architecture documentation according to the taxonomy. We apply different machine learning techniques, such as Logistic Regression, Decision Trees, Random Forests, and BERT to the 17 software architecture documentations. The evaluation yields a F1-score of up to 92.1% for identifying design decisions and a F1-score of up to 55.2% for the classification of the kind of design decision

Detecting Inconsistencies in Software Architecture Documentation Using Traceability Link Recovery

Documenting software architecture is important for a system’s success. Software architecture documentation (SAD) makes information about the system available and eases comprehensibility. There are different forms of SADs like natural language texts and formal models with different benefits and different purposes. However, there can be inconsistent information in different SADs for the same system. Inconsistent documentation then can cause flaws in development and maintenance. To tackle this, we present an approach for inconsistency detection in natural language SAD and formal architecture models. We make use of traceability link recovery (TLR) and extend an existing approach. We utilize the results from TLR to detect unmentioned (i.e., model elements without natural language documentation) and missing model elements (i.e., described but not modeled elements). In our evaluation, we measure how the adaptations on TLR affected its performance. Moreover, we evaluate the inconsistency detection. We use a benchmark with multiple open source projects and compare the results with existing and baseline approaches. For TLR, we achieve an excellent F1-score of 0.81, significantly outperforming the other approaches by at least 0.24. Our approach also achieves excellent results (accuracy: 0.93) for detecting unmentioned model elements and good results for detecting missing model elements (accuracy: 0.75). These results also significantly outperform competing baselines. Although we see room for improvements, the results show that detecting inconsistencies using TLR is promising

Using BERT for the Detection of Architectural Tactics in Code

Quality-driven design decisions are often addressed by using architectural tactics that are re-usable solution options for certain quality concerns. However, it is not suficient to only make good design decisions but also to review the realization of design decisions in code. As manual creation of traceability links for design decisions into code is costly, some approaches perform structural analyses to recover traceability links. However, architectural tactics are high-level solutions described in terms of roles and interactions and there is a wide range of possibilities to implement each. Therefore, structural analyses only yield limited results. Transfer-learning approaches using language models like BERT are a recent trend in the field of natural language processing. These approaches yield state-of-the-art results for tasks like text classification. We intent to experiment with BERT and present an approach to detect architectural tactics in code by fine-tuning BERT. A 10-fold cross-validation shows promising results with an average F1-Score of 90%, which is on a par with state-of-the-art approaches. We additionally apply our approach to a case study, where the results of our approach show promising potential but fall behind the state-of-the-art. Therefore, we discuss our approach and look at potential reasons and downsides as well as potential improvements

NoRBERT: Transfer Learning for Requirements Classification

Classifying requirements is crucial for automatically handling natural language requirements. The performance of existing automatic classification approaches diminishes when applied to unseen projects because requirements usually vary in wording and style. The main problem is poor generalization. We propose NoRBERT that fine-tunes BERT, a language model that has proven useful for transfer learning. We apply our approach to different tasks in the domain of requirements classification. We achieve similar or better results (F$_1$-scores of up to 94%) on both seen and unseen projects for classifying functional and non-functional requirements on the PROMISE NFR dataset. NoRBERT outperforms recent approaches at classifying nonfunctional requirements subclasses. The most frequent classes are classified with an average F$_1$-score of 87%. In an unseen project setup on a relabeled PROMISE NFR dataset, our approach achieves an improvement of ten percentage points in average F$_1$- score compared to recent approaches. Additionally, we propose to classify functional requirements according to the included concerns, i.e., function, data, and behavior. We labeled the functional requirements in the PROMISE NFR dataset and applied our approach. NoRBERT achieves an F$_1$-score of up to 92%. Overall, NoRBERT improves requirements classification and can be applied to unseen projects with convincing results

What’s the Matter? Knowledge Acquisition by Unsupervised Multi-Topic Labeling for Spoken Utterances

Systems such as Alexa, Cortana, and Siri app ear rather smart. However, they only react to predefined wordings and do not actually grasp the user\u27s intent. To overcome this limitation, a system must understand the topics the user is talking about. Therefore, we apply unsupervised multi-topic labeling to spoken utterances. Although topic labeling is a well-studied task on textual documents, its potential for spoken input is almost unexplored. Our approach for topic labeling is tailored to spoken utterances; it copes with short and ungrammatical input. The approach is two-tiered. First, we disambiguate word senses. We utilize Wikipedia as pre-labeled corpus to train a naïve-bayes classifier. Second, we build topic graphs based on DBpedia relations. We use two strategies to determine central terms in the graphs, i.e. the shared topics. One fo cuses on the dominant senses in the utterance and the other covers as many distinct senses as possible. Our approach creates multiple distinct topics per utterance and ranks results. The evaluation shows that the approach is feasible; the word sense disambiguation achieves a recall of 0.799. Concerning topic labeling, in a user study subjects assessed that in 90.9% of the cases at least one proposed topic label among the first four is a good fit. With regard to precision, the subjects judged that 77.2% of the top ranked labels are a good fit or good but somewhat too broad (Fleiss\u27 kappa κ = 0.27). We illustrate areas of application of topic labeling in the field of programming in spoken language. With topic labeling applied to the spoken input as well as ontologies that model the situational context we are able to select the most appropriate ontologies with an F1-score of 0.907

NoRBERT: Transfer Learning for Requirements Classification

Classifying requirements is crucial for automatically handling natural language requirements. The performance of existing automatic classification approaches diminishes when applied to unseen projects because requirements usually vary in wording and style. The main problem is poor generalization. We propose NoRBERT that fine-tunes BERT, a language model that has proven useful for transfer learning. We apply our approach to different tasks in the domain of requirements classification. We achieve similar or better results $\text{F}_1$-scores of up to 94%) on both seen and unseen projects for classifying functional and non-functional requirements on the PROMISE NFR dataset. NoRBERT outperforms recent approaches at classifying non-functional requirements subclasses. The most frequent classes are classified with an average $\text{F}_1$-score of 87%. In an unseen project setup on a relabeled PROMISE NFR dataset, our approach achieves an improvement of 15 percentage points in average $\text{F}_1$-score compared to recent approaches. Additionally, we propose to classify functional requirements according to the included concerns, i.e., function, data, and behavior. We labeled the functional requirements in the PROMISE NFR dataset and applied our approach. NoRBERT achieves an $\text{F}_1$-score of up to 92%. Overall, NoRBERT improves requirements classification and can be applied to unseen projects with convincing results

Establishing a Benchmark Dataset for Traceability Link Recovery between Software Architecture Documentation and Models

In research, evaluation plays a key role to assess the performance of an approach. When evaluating approaches, there is a wide range of possible types of studies that can be used, each with different properties. Benchmarks have the benefit that they establish clearly defined standards and baselines. However, when creating new benchmarks, researchers face various problems regarding the identification of potential data, its mining, as well as the creation of baselines. As a result, some research domains do not have any benchmarks at all. This is the case for traceability link recovery between software architecture documentation and software architecture models. In this paper, we create and describe an open-source benchmark dataset for this research domain. With this benchmark, we define a baseline with a simple approach based on information retrieval techniques. This way, we provide other researchers a way to evaluate and compare their approaches