En este keynote, la Prof. Juristo describe el paradigma experimental y cómo podría aplicarse a la ingeniería del software, destacando los desafíos de su aplicación y los logros conseguidos hasta el momento

Juristo Juzgado, Natalia

Archivo Digital UPM

Applying the experimental paradigm to software engineering

Servicio de Coordinación de Bibliotecas de la Universidad Politécnica de Madrid

Applying  
the Experimental Paradigm  
to Software Engineering  
Natalia Juristo 
Universidad Politécnica de Madrid 
Spain 
 
8th European Computer Science Summit 
Current situation 
n  16.3% of software projects are successful 
 The project is completed on time and within budget, and has all the features 
and functions specified at the start  
 
n  52.7% of software projects cost more, take longer 
or do less 
 The project is completed and operational, but it cost more than budgeted 
(189% more), took longer than estimated and offers fewer features and 
functions than originally specified (42%) 
 
n  31% are cancelled 
 The project is called off at some point during development before the 
system is put into operation 
Current situation 
n  Knowledge 
n  Today the results of applying software development methods are 
unpredictable 
n  There is no evidence to support most of the beliefs on which 
software systems development is based 
n  Practice 
n  Method selection for and decision making on software 
production is based on suppositions and subjective opinions 
n  When, by chance (or thanks to practitioners’ personal and non-
transferable know-how), the right methods are used, the software 
construction projects run smoothly and output the desired product 
n  When the wrong methods are applied, the project develops 
haphazardly and the output product tends to be of poor quality 
Content 
1.  The Scientific Method 
1.  What is science? 
2.  Scientific laws 
3.  Predicting & understanding 
2.  Experimental Software Engineering 
1.  Is the scientific method applicable to SE? 
2.  Experiment & laboratory 
3.  Designing experiments 
4.  Challenges in applying the scientific method to SE 
The Scientific Method 
A process called science 
 
n  Science is a process of understanding the 
world 
n  Science is a way of thinking much more 
than it is a body of knowledge 
 
      Carl Sagan?
Explaining the world 
n  Science looks for explanations about how 
a phenomenon works and why it works as 
we perceive it 
n  These explanations are known as laws or 
theories 
n  Nature generally acts regularly enough to be 
described by laws 
Scientific laws 
n  Are patterns of behaviour 
n  Describe cause-effect relationships 
n  Explain 
n  why some events are related 
n  how the mechanism linking the events 
behaves 
We only perceive nexus 
n  We cannot perceive laws directly 
through our senses 
n  Anyone can see an apple fall, but Newton’s 
inverse-square law of gravitation only becomes 
apparent through special systematic 
measurement 
n  Two activities are necessary 
n  Systematic objective observation 
n  Inference of links between cause & effect 
The scientific method 
n  Collection of empirical data 
n  Systematic observation to appreciate nexus 
n  Theoretical interpretation of data 
n  Formation of a hypotheses (right or wrong) about the 
mechanisms relating the events  
n  Collection of empirical data 
n  Hypotheses need to be tested against reality to find 
out  whether or not they are true?
Building vs. understanding 
n  Humans are able to build interesting 
artefacts without scientific knowledge 
n  The builders do not necessarily 
understand the mechanisms governing the 
observed behaviour 
 
Building without understanding 
Predicting vs. understanding 
 
Empirical  
information     Prediction 
 
 
 
Theoretical      Mechanism 
interpretation     understanding
     
Predicting without understanding 
Data    theory 
n  Without an inferential leap to theory, the 
accumulation of data will never lead to an 
understanding of the mechanism 
n  In the absence of empirical data, theories 
move in the realm of speculation  
n  Not based on arguments from authority or 
popular preferences 
Scientific research method 
n  A rigorous process for properly developing 
and evaluating explanations for 
observable phenomena based on reliable 
empirical evidence and neutral, unbiased 
independent verification 
 
Experimental Software Engineering 
What does all this have to do with 
software? 
Experimental SE 
n  Generates scientific statements about 
building software through experimentation 
n  This knowledge should help to identify the 
applicability conditions, strengths and weaknesses of 
the different software development technologies 
n  All engineering disciplines have taken a 
similar step  
n  Achieve predictable results moving from beliefs, 
speculations and lucky guesses to scientific 
knowledge 
Gaining SE knowledge with 
experiments 
n  Identify and understand 
n  the variables that play a role in software development 
n  the connections between variables 
  
n  Learn cause-effect relationships between the 
development process and the resulting products 
n  Establish laws and theories about software 
construction that explain development behaviour 
Experimental Software Engineering 
Experiment & Laboratory 
Experiment 
n  Experiments 
n  Model key characteristics of a reality in a controlled 
environment by manipulating them iteratively to 
investigate the impact of such variations and get a 
better understanding of a phenomenon 
n  Are a formal, rigorous and controlled investigation in 
which the variables under study are given different 
values to find out what effect each value has 
n  The properties of a complex system are 
explained by analysing the behaviour of its parts 
SE experiment 
n  Development decomposed into its parts 
n  Manipulated variables 
n  Techniques (design, testing, etc.),  
n  Developers (experience, knowledge, etc.) 
n  Variables that can be assigned during development 
 
n  Investigated impacts 
n  Effectiveness, efficiency, productivity, quality 
n  Examples of instances 
n  number of detected defects, number of code lines, etc. 
n  Interesting characteristics obtained as a result of development 
The laboratory 
n  Laboratory  
n  Simplified and controllable reality where the phenomenon under 
study can be manipulated and studied 
n  Chemistry laboratory  
n  Flasks and pipettes where temperatures and pressures are 
controlled 
n  Real world: real substances with temperature and pressures 
n  Economics laboratory 
n  Sets of individuals playing games to earn toy benefits 
n  Real-world: markets (composed of thousands of agents) where 
real rewards are pursued 
n  What is a SE laboratory like?? 
SE laboratory 
n  Students  
n  rather than professionals  
n  Toy software  
n  rather than real systems 
n  Exercises  
n  rather than real projects 
n  Academic workshops or industrial tutorials 
n  rather than real knowledge & experience in industry 
n  Phases, techniques  
n  rather than whole projects 
Weakness 
n  How representative is any lab finding of 
reality? 
n  Different levels of experimental studies  
n  In vitro experiments 
n  In vivo experiments (from mice to monkeys) 
n  Field experiments (from volunteers to clinical trials) 
 
External validity    
n  Is concerned with the extent to which the 
results can be generalized  
n  from the unique and idiosyncratic 
experimental settings, procedures and 
participants  
n  to other populations and conditions 
n  Generalizability of experimental results to 
n  the target population of the study  
n  the universe of other populations  
Experimental Software Engineering 
Designing Experiments 
Cause-effect relationships 
n  The independent variable 
n  is the variable that is thought to be the cause  
n  must meet two requirements 
n  be changeable  
n  the change must be controllable 
n  The dependent variable is  
n  the effect brought about by this cause 
n  is not manipulated 
n  measured to see how it is affected by the 
manipulation of the independent variable 
 
Extraneous variables 
n  If extraneous variables also vary systematically 
with the independent variables 
n  then conclusions regarding causality are not valid 
n  the observations are “confounded” 
n  Experiment design involves controlling the 
influence of extraneous variables on the 
dependent variables 
n  Good design avoids confounding variables 
 
Control strategies 
n  Control neutralizes variation of extraneous variables 
n  Control strategies 
n  Constancy 
n  Keeping extraneous variables constant 
n  Blocking 
n  Neutralizing known extraneous variables 
n  Purposely assigning every value of the blocked variable to every 
group 
n  Randomization 
n  Neutralizing unknown extraneous variables 
n  Random assignment of subjects to experiment conditions 
Experiment design 
n  The validity of the design of experiments is a 
fundamental part of the scientific method 
n  The design of a controlled experiment is a set of 
strategies aiming to control 
n  Without a valid design, valid conclusions cannot 
be drawn 
n  Statistics cannot fix a badly designed 
experiment 
Internal validity    
n  Is concerned with whether we can 
accurately infer that  
n  the independent variable caused the effect on 
the dependent variable 
n  Certainty with which we can establish the 
cause of the variations in results 
n  Were there any extraneous variables that 
could have caused the observed effect? 
Experimental Software Engineering 
Current Status 
 
Journals, conferences, books 
n  A specialized journal  
n  Empirical Software Engineering Journal 
n  A specialized conference 
n  Empirical SE and Measurement Conference 
n  A couple of books 
n  Experimentation in SE: An Introduction  
 Wohlin, Runeson, Höst, Ohlsson, Regnell, Wesslén 
 Springer 2012 
 
n  Basics of SE Experimentation  
 Juristo & Moreno  
 Springer 2001 
  
 
ESE: Origins 
n  The first experiments were run in the early 1980s by 
Victor Basili’s group at the University of Maryland with 
NASA’s Software Engineering Laboratory 
n  The use of experiments to examine the applicability of 
SE technologies has gradually gained in importance as 
a research methodology 
n  Empirical studies have finally become recognized as an 
important component of the SE discipline 
n  Fraction of empirical studies is rising in the last 3-4 years 
ESE: Evolution 
n  1993-2003: Leading SE journals  
n  TSE, TOSEM, JSS,  EMSE, IST, IEEE Software, IEEE 
Computer, SP&E 
n  78 experiments 
n  1977-2006: ICSE 
n  3.2% had some type of empirical evaluation. Of such 
n  0.9% case studies; 0% experiments; 0.7% quasi-experiments 
n  0% the contribution was pure empirical 
n  2012: ICSE  
n  71% had an empirical evaluation 
n  20% the contribution was pure empirical  
n  experiments, case studies & qualitative 
SE Experimental paradigm  
     is still immature 
n  SE experiments are mostly exploratory 
n  They produce objective observation but 
cannot yet be explained 
n  The scientific method’s inference step is not 
being exercised  
n  Finding (statistical) patterns is not enough 
n  Mechanisms have to be found that explain the 
patterns 
SE Experimental paradigm  
     is still immature 
n  SE experiments have flaws 
n  Lack of thoroughly thought-out designs to rule 
out extraneous variables in each experiment  
n  Proper analysis techniques are not always 
used  
n  Lack of replications 
n  Lack of field experiments 
It is not Just Import the Paradigm 
n  ESE lays the foundations for carrying out 
experimental research into SE 
n  It is not enough just to apply experimental 
design and statistical data analysis to take 
experimental research in SE forward 
n  A discipline’s specific experimental 
methodology cannot be imported directly from 
others  
 
THE ROLE OF  
SCIENTIFIC METHOD IN 
SOFTWARE DEVELOPMENT 
Natalia Juristo 
Universidad Politécnica de Madrid 
Spain 


http://oa.upm.es/20331/1/INVE_MEM_2012_133333.pdf

Applying the experimental paradigm to software engineering

Abstract

Similar works

Full text

Available Versions

Archivo Digital UPM

Servicio de Coordinación de Bibliotecas de la Universidad Politécnica de Madrid