Proposal of a Quasi-Experiment for studying the effect of experience on elicitation effectiveness

We plan to perform a quasi experiment to evaluate the effect of experience on requirements elicitation. Researchers will play the role of customers, whereas participants will perform the role of analysts. Analysts will hold a 60 minute interview and will then be given 25 minutes to write up a report of their findings. Participant effectiveness will be compared with available data series on the effectiveness of novice analysts that we have collected previously

Evidence of the presence of bias in subjective metrics: analysis within a family of experiments

Context: Measurement is crucial and important to empirical software engineering. Although reliability and validity are two important properties warranting consideration in measurement processes, they may be influenced by random or systematic error (bias) depending on which metric is used. Aim: Check whether, the simple subjective metrics used in empirical software engineering studies are prone to bias. Method: Comparison of the reliability of a family of empirical studies on requirements elicitation that explore the same phenomenon using different design types and objective and subjective metrics. Results: The objectively measured variables (experience and knowledge) tend to achieve more reliable results, whereas subjective metrics using Likert scales (expertise and familiarity) tend to be influenced by systematic error or bias. Conclusions: Studies that predominantly use variables measured subjectively, like opinion polls or expert opinion acquisition

Effect of domain knowledge on elicitation effectiveness: an internally replicated controlled experiment

Context. Requirements elicitation is a highly communicative activity in which human interactions play a critical role. A number of analyst characteristics or skills may influence elicitation process effectiveness. Aim. Study the influence of analyst problem domain knowledge on elicitation effectiveness. Method. We executed a controlled experiment with post-graduate students. The experimental task was to elicit requirements using open interview and consolidate the elicited information immediately afterwards. We used four different problem domains about which students had different levels of knowledge. Two tasks were used in the experiment, whereas the other two were used in an internal replication of the experiment; that is, we repeated the experiment with the same subjects but with different domains. Results. Analyst problem domain knowledge has a small but statistically significant effect on the effectiveness of the requirements elicitation activity. The interviewee has a big positive and significant influence, as does general training in requirements activities and interview experience. Conclusion. During early contacts with the customer, a key factor is the interviewee; however, training in tasks related to requirements elicitation and knowledge of the problem domain helps requirements analysts to be more effectiv

The Impact of Domain Knowledge on the Effectiveness of Requirements Engineering Activities

One of the factors that seems to influence an individual’s effectiveness in requirements engineering activities is his or her knowledge of the problem being solved, i.e., domain knowledge. While in-depth domain knowledge enables a requirements engineer to understand the problem easier, he or she can fall for tacit assumptions of the domain and might overlook issues that are obvious to domain experts and thus remain unmentioned. The purpose of this thesis is to investigate the impact of domain knowledge on different requirements engineering activities. The main research question this thesis attempts to answer is “How does one form the most effective team, consisting of some mix of domain ignorants and domain awares, for a requirements engineering activity involving knowledge about the domain of the computer-based system whose requirements are being determined by the team?” This thesis presents two controlled experiments and an industrial case study to test a number of hypotheses. The main hypothesis states that a requirements engineering team for a computer-based system in a particular domain, consisting of a mix of requirements analysts that are ignorant of the domain and requirements analysts that are aware of the domain, is more effective at requirement idea generation than a team consisting of only requirements analysts that are aware of the domain. The results of the controlled experiments, although not conclusive, provided some support for the positive effect of the mix on effectiveness of a requirements engineering team. The results also showed a significant effect of other independent variables, especially educational background. The data of the case study corroborated the results of the controlled experiments. The main conclusion that can be drawn from the findings of this thesis is that the presence in a requirements engineering team of a domain ignorant with a computer science or software engineering background improves the effectiveness of the team

Domain- and Quality-aware Requirements Engineering for Law-compliant Systems

Titel in deutscher Übersetzung: Domänen- und qualitätsgetriebene Anforderungserhebung für gesetzeskonforme Systeme Der bekannte Leitsatz in der Anforderungserhebung und -analyse besagt, dass es schwierig ist, das richtige System zu bauen, wenn man nicht weiß, was das 'Richtige' eigentlich ist. Es existieren überzeugende Belege, dass dieser Leitsatz die Notwendigkeit der Anforderungserhebung und -analyse exakt definiert und beschreibt. Zum Beispiel ergaben Studien, dass das Beheben von Defekten in einer Software, die bereits produktiv genutzt wird, bis zu 80 mal so teuer ist wie das frühzeitige Beheben der korrespondierenden Defekte in den Anforderungen. Generell hat es sich gezeigt, dass das Durchführen einer angemessenen Anforderungserhebung und -analyse ein wichtiger Erfolgsfaktor für Softwareentwicklungsprojekte ist. Während der Progression von den initialen Wünschen der beteiligten Interessensvertretern für ein zu entwickelndes System zu einer Spezifikation für eben dieses Systems müssen Anforderungsanalysten einen komplexen Entscheidungsprozess durchlaufen, der die initialen Wünsche in die Spezifikation überführt. Tatsächlich wird das Treffen von Entscheidungen als integraler Bestandteil der Anforderungsanalyse gesehen. In dieser Arbeit werden wir versuchen zu verstehen welche Aktivitäten und Information von Nöten sind, um eine fundierte Auswahl von Anforderungen vorzunehmen, welche Herausforderungen damit verbunden sind, wie eine ideale Lösung zur Anforderungswahl aussehen könnte und in welchen Bereichen der aktuelle Stand der Technik in Bezug auf diese ideale Lösung lückenhaft ist. Innerhalb dieser Arbeit werden wir die Informationen, die notwendig für eine fundierte Anforderungsauswahl sind, identifizieren, einen Prozess präsentieren, um diese notwendigen Informationen zu sammeln, die Herausforderungen herausstellen, die durch diesen Prozess und die damit verbundenen Aktivitäten adressiert werden und eine Auswahl von Methoden diskutieren, mit deren Hilfe man die Aktivitäten des Prozesses umsetzen kann. Die gesammelten Informationen werden dann für eine automatisierte Anforderungsauswahl verwendet. Für die Auswahl kommt ein Optimierungsmodell, das Teil des Beitrags dieser Arbeit ist, zum Einsatz. Da wir während der Erstellung dieser Arbeit zwei große Lücken im Stand der Technik bezüglich unseres Prozesses und der damit verbundenen Aktivitäten identifiziert haben, präsentieren wir darüber hinaus zwei neuartige Methoden für die Kontexterhebung und die Erhebung von rechtlichen Anforderungen, um diese Lücken zu schließen. Diese Methoden sind Teil des Hauptbeitrags dieser Arbeit. Unsere Lösung für der Erhebung des Kontext für ein zu entwickelndes System ermöglicht das Etablieren eines domänenspezifischen Kontextes unter Zuhilfenahme von Mustern für verschiedene Domänen. Diese Kontextmuster erlauben eine strukturierte Erhebung und Dokumentation aller relevanten Interessensvertreter und technischen Entitäten für ein zu entwickelndes System. Sowohl die Dokumentation in Form von grafischen Musterinstanzen und textuellen Vorlageninstanzen als auch die Methode zum Sammeln der notwendigen Informationen sind expliziter Bestandteil jedes Kontextmusters. Zusätzlich stellen wir auch Hilfsmittel für die Erstellung neuer Kontextmuster und das Erweitern der in dieser Arbeit präsentierten Kontextmustersprache zur Verfügung. Unsere Lösung für die Erhebung von rechtlichen Anforderungen basiert auch auf Mustern und stellt eine Methode bereit, welche es einem erlaubt, die relevanten Gesetze für ein zu erstellendes System, welches in Form der funktionalen Anforderungen bereits beschrieben sein muss, zu identifizieren und welche die bestehenden funktionalen Anforderungen mit den rechtlichen Anforderungen verknüpft. Diese Methode beruht auf der Zusammenarbeit zwischen Anforderungsanalysten und Rechtsexperten und schließt die Verständnislücke zwischen ihren verschiedenartigen Welten. Wir veranschaulichen unseren Prozess unter der Zuhilfenahme eines durchgehenden Beispiels aus dem Bereich der service-orientierten Architekturen. Zusätzlich präsentieren wir sowohl die Ergebnisse der Anwendung unseres Prozesses (bzw. Teilen davon) auf zwei reale Fälle aus den Bereichen von Smart Grids und Wahlsystemen, als auch alle anderen Ergebnisse der wissenschaftlichen Methoden, die wir genutzt haben, um unsere Lösung zu fundieren und validieren.The long known credo of requirements engineering states that it is challenging to build the right system if you do not know what right is. There is strong evidence that this credo exactly defines and describes the necessity of requirements engineering. Fixing a defect when it is already fielded is reported to be up to eighty times more expensive than fixing the corresponding requirements defects early on. In general, conducting sufficient requirements engineering has shown to be a crucial success factor for software development projects. Throughout the progression from initial stakeholders' wishes regarding the system-to-be to a specification for the system-to-be requirements engineers have to undergo a complex decision process for forming the actual plan connecting stakeholder wishes and the final specification. Indeed, decision making is considered to be an inherent part of requirements engineering. In this thesis, we try to understand which activities and information are needed for selecting requirements, which the challenges are, how an ideal solution for selecting requirements would look like, and where the current state of the art is deficient regarding the ideal solution. Within this thesis we identify the information necessary for an informed requirements selection, present a process in which one collects all the necessary information, highlight the challenges to be addressed by this process and its activities, and a selection of methods to conduct the activities of the process. All the collected information is then used for an automated requirements selection using an optimization model which is also part of the contribution of this thesis. As we identified two major gaps in the state of the art considering the proposed process and its activities, we also present two novel methods for context elicitation and for legal compliance requirements elicitation to fill the gaps as part of the main contribution. Our solution for context elicitation enables a domain-specific context establishment based on patterns for different domains. The context patterns allow a structured elicitation and documentation of relevant stakeholders and technical entities for a system-to-be. Both, the documentation in means of graphical pattern instances and textual template instances as well as the method for collecting the necessary information are explicitly given in each context pattern. Additionally, we also provide the means which are necessary to derive new context patterns and extend our context patterns language which is part of this thesis. Our solution for legal compliance requirements elicitation is a pattern-based and guided method which lets one identify the relevant laws for a system-to-be, which is described in means of functional requirements, and which intertwines the functional requirements with the according legal requirements. This method relies on the collaboration of requirements engineers and legal experts, and bridges the gap between their distinct worlds. Our process is exemplified using a running example in the domain of service oriented architectures. Additionally, the results of applying (parts of) the process to real life cases from the smart grid domain and voting system domain are presented, as well as all other results from the scientific means we took to ground and validate the proposed solutions