CRAFT: Concept Recursive Activation FacTorization for Explainability

Attribution methods, which employ heatmaps to identify the most influential regions of an image that impact model decisions, have gained widespread popularity as a type of explainability method. However, recent research has exposed the limited practical value of these methods, attributed in part to their narrow focus on the most prominent regions of an image -- revealing "where" the model looks, but failing to elucidate "what" the model sees in those areas. In this work, we try to fill in this gap with CRAFT -- a novel approach to identify both "what" and "where" by generating concept-based explanations. We introduce 3 new ingredients to the automatic concept extraction literature: (i) a recursive strategy to detect and decompose concepts across layers, (ii) a novel method for a more faithful estimation of concept importance using Sobol indices, and (iii) the use of implicit differentiation to unlock Concept Attribution Maps. We conduct both human and computer vision experiments to demonstrate the benefits of the proposed approach. We show that the proposed concept importance estimation technique is more faithful to the model than previous methods. When evaluating the usefulness of the method for human experimenters on a human-centered utility benchmark, we find that our approach significantly improves on two of the three test scenarios. Our code is freely available at github.com/deel-ai/Craft

WHY HASN\u27T STUDYING PERCEPTION IN AUTISM SPECTRUM DISORDERS HELPED US CREATE A COGNITIVE MODEL?

There are a number of cognitive models of autism that aim to explain how mental processes are handled differently in the condition. These models make claims about the nature of cognitive function in people with autism, and suggest that these differences applied in social contexts lead to the characteristic behavioural patterns. However, it is difficult to study these cognitive differences directly because of the complexity of social situations. Studies of perceptual function are tempting as an alternative way to study cognition because it is far easier to control the conditions and the stimuli that participants are exposed to. This makes hypothesis generation and interpretation of results more objective and more convincing. However, the study of perception in autism hasn\u27t been very productive in contributing towards a model of cognition in autism. In many areas there are studies reporting contradictory results, preventing arrival at a consensus about the largest unresolved issues in the area. These studies tend to be repeated multiple times, but continue to provide contradictory evidence that doesn\u27t allow us to place confidence in any of the cognitive models. An approach to these issues is proposed, focusing on critical analysis of contradictory studies rather than the endless process of repetition. This allows previous studies to be interpreted more objectively and resolve conflicts, and guides the design of future studies in ways that avoid the pitfalls that have been identified. Both of these outcomes result in more productive work being done. The first example is in the study of motion perception in autism, where the use of non-identical stimuli has been problematic. On closer critical analysis, a fundamental aspect of the motion stimuli demonstrates that the contradictions might be expected based on the differences in stimuli used. Addressing this issue can move the field towards resolution. A second example is in the study of spatial frequency sensitivity. Here, poor study design has created results leading to an "eagle-eyed visual acuity" hypothesis of autism. Errors in the initial study are revealed, suggesting that the model should be abandoned. Finally, a general issue is the assumption of homogeneity of perceptual ability and genetics in autism, where the reality is that subgroups exist within the population of people with autism, and significant variation exists between them. The evidence for this is summarised and the issues that it creates explored

Psychophysics with children: Evaluating the use of maximum likelihood estimators in children aged 4-15 years (QUEST plus )

Maximum Likelihood (ML) estimators such as QUEST+ allow complex psychophysical measurements to be made more quickly and precisely than traditional staircase techniques. They could therefore be useful for quantifying sensory function in populations with limited attention spans, such as children. To test this, the present study empirically evaluated the performance of an ML estimator (QUEST+) versus a traditional Up-Down Weighted Staircase in children and adults. Seventy-one children (4.7–14.7 years) and 43 adults (18.1–29.6 years) completed a typical psychophysical procedure: Contrast Sensitivity Function (CSF) determination. Some participants were tested twice with the same method (QUEST+ or Staircase), allowing test-retest repeatability to be quantified. Others were tested once each with either method (QUEST+ and Staircase), allowing accuracy to be quantified. The results showed that QUEST+ was more efficient: In both children and adults, approximately half the number of ML trials were required to attain comparable levels of accuracy and reliability as a traditional Staircase paradigm, and plausible CSF estimates could be made in even the youngest children. The ML procedure was also as robust as the Staircase to lapses in concentration, and its performance did not depend on prespecifying correct model priors. The results show that ML estimators could greatly improve our ability to study sensory processes and detect impairments in children, although important practical considerations for-and-against their use are discussed

What Makes Teachers Effective: Investigating the Relationship Between CABAS® Teacher Ranks and Teacher Effectiveness

I examined the relationship between teacher effectiveness as measured by the number of learn units students required to meet an objective and the number of competencies mastered within the categories of teacher repertoires composing the CABAS® rank. Twenty preschool teachers participated in the study. A statistical analysis was used to investigate the degree to which these variables negatively correlated with each other. The results showed that the more competencies teachers mastered, the fewer learn units students required to meet an objective. A second experiment was conducted as an experimental analysis of the correlations found in the descriptive analysis. An adapted alternating treatments design was used to analyze the relationship between the number of competencies teachers mastered and the number of learn units their student required to meet an objective. Four teachers and four teacher assistants participated in the study. The teachers and teacher assistants each taught two sight word objectives for a student with bidirectional naming and a student without bidirectional naming. The results did not show a functional relationship between the number of competencies mastered and a lower LUC (learn unit to criterion). Teachers with more competencies mastered did not present fewer learn units for their students to meet an objective when compared to teacher assistants who had fewer competencies mastered. Possible explanations for a lack of a functional relationship found in Experiment 2 are discussed

Video feedback intervention: a case series in the context of childhood hearing impairment

Background: Recent research shows that parental sensitivity can explain a significant and unique amount of growth in speech and language outcomes in children with cochlear implants. In this intervention study we explored the impact of an intervention designed to support parental sensitivity on children's communication development. Aims: This study tests the effect of a complex intervention in the context of childhood hearing impairment using a case study design of three families. Propositions for each case were made using parental report of the child's development in an attempt to identify change in outcome measurements that were not likely to be due to general development in the child or a halo effect from the intervention. Methods and Results: Multiple pre- and post-intervention measures were taken. Outcome measures were mother–child contingencies to vocal utterances, emotional availability and an assessment of early communication in the child. Results for each case showed that improvements in some outcome measurements were found after the intervention and were maintained at follow-up. Conclusions & Implications: Taking account of developmental change in intervention studies with children is challenging. Single-subject intervention studies can be designed to allow research interventions to be tailored to meet families’ specific needs. Video interaction guidance may support pre-linguistic communicative development in children with hearing impairment

Online experimentation in automotive software engineering

Context: Online experimentation has long been the gold standard for evaluating software towards the actual needs and preferences of customers. In the Software-as-a-Service domain, various online experimentation techniques are applied and proven successful. As software is becoming the main differentiator for automotive products, the automotive sector has started to express an interest in adopting online experimentation to strengthen their software development process. Objective: In this research, we aim to systematically address the challenges in adopting online experimentation in the automotive domain.Method: We apply a multidisciplinary approach to this research. To understand the state-of-practise in online experimentation in the industry, we conduct case studies with three manufacturers. We introduce our experimental design and evaluation methods to real vehicles driven by customers at scale. Moreover, we run experiments to quantitatively evaluate experiment design and causal inference models. Results: Four main research outcomes are presented in this thesis. First, we propose an architecture for continuous online experimentation given the limitations experienced in the automotive domain. Second, after identifying an inherent limitation of sample sizes in the automotive domain, we apply and evaluate an experimentation design method. The method allows us to utilise pre-experimental data for generating balanced groups even when sample sizes are limited. Third, we present an alternative approach to randomised experiments and demonstrate the application of Bayesian causal inference in online software evaluation. With the models, we enable software online evaluation without the need for a fully randomised experiment. Finally, we relate the formal assumption in the Bayesian causal models to the implications in practise, and we demonstrate the inference models with cases from the automotive domain. Outlook: In our future work, we plan to explore causal structural and graphical models applied in software engineering, and demonstrate the application of causal discovery in machine learning-based autonomous drive software