Making inferences with small numbers of training sets

A potential methodological problem with empirical studies that assess project effort prediction system is discussed. Frequently, a hold-out strategy is deployed so that the data set is split into a training and a validation set. Inferences are then made concerning the relative accuracy of the different prediction techniques under examination. This is typically done on very small numbers of sampled training sets. It is shown that such studies can lead to almost random results (particularly where relatively small effects are being studied). To illustrate this problem, two data sets are analysed using a configuration problem for case-based prediction and results generated from 100 training sets. This enables results to be produced with quantified confidence limits. From this it is concluded that in both cases using less than five training sets leads to untrustworthy results, and ideally more than 20 sets should be deployed. Unfortunately, this raises a question over a number of empirical validations of prediction techniques, and so it is suggested that further research is needed as a matter of urgency

How reliable are systematic reviews in empirical software engineering?

BACKGROUND – the systematic review is becoming a more commonly employed research instrument in empirical software engineering. Before undue reliance is placed on the outcomes of such reviews it would seem useful to consider the robustness of the approach in this particular research context. OBJECTIVE – the aim of this study is to assess the reliability of systematic reviews as a research instrument. In particular we wish to investigate the consistency of process and the stability of outcomes. METHOD – we compare the results of two independent reviews under taken with a common research question. RESULTS – the two reviews find similar answers to the research question, although the means of arriving at those answers vary. CONCLUSIONS – in addressing a well-bounded research question, groups of researchers with similar domain experience can arrive at the same review outcomes, even though they may do so in different ways. This provides evidence that, in this context at least, the systematic review is a robust research method

Teacher Supply and Demand: Issues in Northern Canada

This two‐year study (2007‐2009), which examined teacher supply and demand issues in northern Canada – Fort Nelson School District (BC), the Fort Vermilion School Division (AB), the Yukon Department of Education (YK), and the Yellowknife School District (NWT) – comprised three research objectives: (a) to ascertain in which subject areas acute and chronic needs for teachers existed, (b) to investigate recruitment and retention methods for northern professionals, and (c) to ascertain preferred professional development models. The participants included teachers, principals, and hiring personnel in the research sites. Research methods included the use of an on‐line questionnaire (n = 113), at least two semi‐structured interviews at each of the four sites with five to ten teachers, three to five principals, and one Human Resources personnel, and researcher field notes. Findings confirm and expand upon conclusions drawn by others and demonstrate the unique needs of northern educators. In particular, we found that (a) school districts continue to struggle with finding specialist teachers inthe senior high sciences and mathematics and in elementary Special Education, (b) few incentives exist for teachers to come to northern school districts and to stay there, and (c) the preferred professional development model for approximately one‐quarter of these Northern teachers, administrators, and hiring personnel is blended learning