Just because software developers say they believe in "X", that does not
necessarily mean that "X" is true. As shown here, there exist numerous beliefs
listed in the recent Software Engineering literature which are only supported
by small portions of the available data. Hence we ask what is the source of
this disconnect between beliefs and evidence?. To answer this question we look
for evidence for ten beliefs within 300,000+ changes seen in dozens of
open-source projects. Some of those beliefs had strong support across all the
projects; specifically, "A commit that involves more added and removed lines is
more bug-prone" and "Files with fewer lines contributed by their owners (who
contribute most changes) are bug-prone". Most of the widely-held beliefs
studied are only sporadically supported in the data; i.e. large effects can
appear in project data and then disappear in subsequent releases. Such sporadic
support explains why developers believe things that were relevant to their
prior work, but not necessarily their current work. Our conclusion will be that
we need to change the nature of the debate with Software Engineering.
Specifically, while it is important to report the effects that hold right now,
it is also important to report on what effects change over time.Comment: 9 pages, 3 Figures, 4 Tables, ICSE SEIP 202