A Zero-Positive Learning Approach for Diagnosing Software Performance Regressions

The field of machine programming (MP), the automation of the development of software, is making notable research advances. This is, in part, due to the emergence of a wide range of novel techniques in machine learning. In this paper, we apply MP to the automation of software performance regression testing. A performance regression is a software performance degradation caused by a code change. We present AutoPerf–a novel approach to automate regression testing that utilizes three core techniques:(i) zero-positive learning,(ii) autoencoders, and (iii) hardware telemetry. We demonstrate AutoPerf’s generality and efficacy against 3 types of performance regressions across 10 real performance bugs in 7 benchmark and open-source programs. On average, AutoPerf exhibits 4% profiling overhead and accurately diagnoses more performance bugs than prior state-of-the-art approaches. Thus far, AutoPerf has produced no false negatives

ExplainIt! -- A declarative root-cause analysis engine for time series data (extended version)

We present ExplainIt!, a declarative, unsupervised root-cause analysis engine that uses time series monitoring data from large complex systems such as data centres. ExplainIt! empowers operators to succinctly specify a large number of causal hypotheses to search for causes of interesting events. ExplainIt! then ranks these hypotheses, reducing the number of causal dependencies from hundreds of thousands to a handful for human understanding. We show how a declarative language, such as SQL, can be effective in declaratively enumerating hypotheses that probe the structure of an unknown probabilistic graphical causal model of the underlying system. Our thesis is that databases are in a unique position to enable users to rapidly explore the possible causal mechanisms in data collected from diverse sources. We empirically demonstrate how ExplainIt! had helped us resolve over 30 performance issues in a commercial product since late 2014, of which we discuss a few cases in detail.Comment: SIGMOD Industry Track 201

The unexplained nature of reading.

The effects of properties of words on their reading aloud response times (RTs) are 1 major source of evidence about the reading process. The precision with which such RTs could potentially be predicted by word properties is critical to evaluate our understanding of reading but is often underestimated due to contamination from individual differences. We estimated this precision without such contamination individually for 4 people who each read 2,820 words 50 times each. These estimates were compared to the precision achieved by a 31-variable regression model that outperforms current cognitive models on variance-explained criteria. Most (around 2/3) of the meaningful (non-first-phoneme, non-noise) word-level variance remained unexplained by this model. Considerable empirical and theoretical-computational effort has been expended on this area of psychology, but the high level of systematic variance remaining unexplained suggests doubts regarding contemporary accounts of the details of the mechanisms of reading at the level of the word. Future assessment of models can take advantage of the availability of our precise participant-level database

Math anxiety, intrusive thoughts and performance: Exploring the relationship between mathematics anxiety and performance: The role of intrusive thoughts

The current study examined the relationship between math anxiety and arithmetic performance by focusing on intrusive thoughts experienced during problem solving. Participants (N = 122) performed two-digit addition problems on a verification task. Math anxiety significantly predicted response time and error rate. Further, the extent to which intrusive thoughts impeded calculation mediated the relationship between math anxiety and per cent of errors on problems involving a carry operation. Moreover, results indicated that participants experienced a range of intrusive thoughts and these were related to significantly higher levels of math anxiety. The findings lend support to a deficient inhibition account of the math anxiety-to-performance relationship and highlight the importance of considering intrusive thoughts in future work

The Relative Performance of Targeted Maximum Likelihood Estimators

There is an active debate in the literature on censored data about the relative performance of model based maximum likelihood estimators, IPCW-estimators, and a variety of double robust semiparametric efficient estimators. Kang and Schafer (2007) demonstrate the fragility of double robust and IPCW-estimators in a simulation study with positivity violations. They focus on a simple missing data problem with covariates where one desires to estimate the mean of an outcome that is subject to missingness. Responses by Robins et al. (2007), Tsiatis and Davidian (2007), Tan (2007a) and Ridgeway and McCaffrey (2007) further explore the challenges faced by double robust estimators and offer suggestions for improving their stability. In this article, we join the debate by presenting targeted maximum likelihood estimators (TMLEs). We demonstrate that TMLEs that guarantee that the parametric submodel employed by the TMLE-procedure respects the global bounds on the continuous outcomes, are especially suitable for dealing with positivity violations because in addition to being double robust and semiparametric efficient, they are substitution estimators. We demonstrate the practical performance of TMLEs relative to other estimators in the simulations designed by Kang and Schafer (2007) and in modified simulations with even greater estimation challenges

Enhancing pre-service teachers' diagnostic competence in Physics misconceptions at public universities in Tanzania

Teachers require to diagnose students’ learning needs in order to plan and carryout effective lessons, a process similar to medical doctors diagnosing their patients before treatment. While it is crucial to enhance diagnostic competence in teachers, an issue remains about how we can best improve this competence among undergraduate pre-service teachers. In the teaching and learning process of science in middle or high schools, misconceptions can hinder learning of new Physics ideas if teachers do not detect and correct them in time. The current research carried out a meta-analysis of 22 empirical studies aimed at fostering diagnostic competences through intervention in teacher and medical education, summarized the findings, revealed the overall effect size, and examined the moderating factors. Following the results of the meta-analysis, we designed an experimental study to investigate the effects of problem solving and example-based learning instructional approaches on enhancing pre-service teachers’ diagnostic competence in Physics misconceptions. The meta-analysis revealed a positive medium mean effect size (g = 0.37) of interventions on fostering the development of diagnostic competences among undergraduate students in both domains. The moderator analysis suggests that an instructional approach is a significant moderator when we apply problem solving during the learning phase of an intervention. The experimental study revealed that both problem solving and example-based learning significantly enhanced pre-service teachers’ diagnostic competence in form of conceptual knowledge, but not the procedural knowledge. Problem solving instructional approach was more effective than example-based learning on enhancing diagnostic competence. The pre-service teachers’ diagnostic competence in the form of conceptual and procedural knowledge positively correlated with germane cognitive load, while it negatively correlated with intrinsic and extraneous cognitive loads. Example-based learning instructional approach significantly influenced both intrinsic and extraneous cognitive loads when compared with problem solving. Cognitive load did not significantly mediate the effect of the instructional approaches on diagnostic competences, and a rating scale questionnaire differentiated between the three types of cognitive load, but did not clearly discriminate between intrinsic and extraneous cognitive loads. The meta-analysis findings imply that learning to diagnose various aspects through problem solving is an effective means of advancing undergraduate students’ diagnostic competences. Learners’ prior diagnostic knowledge seems to be a covariate on enhancing diagnostic competences through interventions. An experimental study findings also imply that the problem solving instructional approach can enhance pre-service teacher’s diagnostic competence in identifying pupil’s Physics misconceptions better than example-based learning. In practice, the current research supports the assumption that integrating diagnostic practices into the Physics-methods course curriculum during undergraduate training programs can improve pre-service Physics teachers’ formative assessment skills. Some limitations can be accounted for by the findings in both studies. With respect to the meta-analysis, the restrictions of robust variance estimation method when estimating meta-regressions especially for moderator analyses could have limited the findings due to imbalances of level of some categorical moderator variables. This could have then affected the degrees of freedom and hence the power for moderation effect. In the experimental study, the random errors that might occur due to extraneous variable (e.g. individual ability) that could have affected the outcome measures rather than intervention treatment, and the assessment of pre-service teachers’ diagnostic knowledge through a same knowledge test could have also limited the findings. In conclusion, the meta-analysis supports the development of diagnostic competence through interventions (with a medium effect size), and indicates that problem solving is the best instructional approach. The meta-analysis also seems to point out the fact that example-based learning instructional approach may better fit learners with lower prior knowledge, whereas, problem solving may better fit learners with higher levels of prior knowledge. With respect to the experimental study, undergraduate pre-service teachers seem to learn abstract concepts and ideas about the diagnosis process better through problem solving than example-based learning. Both instructional approaches seem to facilitate the diagnostic competence effectively, if we consider the germane cognitive load high, while keeping the intrinsic and cognitive load to a minimum. The current research further emphasizes the need for a similar meta-analysis to include more studies and alternative moderators (e.g. types of feedback, prompts, and so on), and an experimental study to compare the effects of problem solving and example-based learning on diagnostic competences with immediate and delayed post testing

Performance Problem Diagnostics by Systematic Experimentation

Diagnostics of performance problems requires deep expertise in performance engineering and entails a high manual effort. As a consequence, performance evaluations are postponed to the last minute of the development process. In this thesis, we introduce an automatic, experiment-based approach for performance problem diagnostics in enterprise software systems. With this approach, performance engineers can concentrate on their core competences instead of conducting repeating tasks