Revisiting Precision and Recall Definition for Generative Model Evaluation

In this article we revisit the definition of Precision-Recall (PR) curves for generative models proposed by Sajjadi et al. (arXiv:1806.00035). Rather than providing a scalar for generative quality, PR curves distinguish mode-collapse (poor recall) and bad quality (poor precision). We first generalize their formulation to arbitrary measures, hence removing any restriction to finite support. We also expose a bridge between PR curves and type I and type II error rates of likelihood ratio classifiers on the task of discriminating between samples of the two distributions. Building upon this new perspective, we propose a novel algorithm to approximate precision-recall curves, that shares some interesting methodological properties with the hypothesis testing technique from Lopez-Paz et al (arXiv:1610.06545). We demonstrate the interest of the proposed formulation over the original approach on controlled multi-modal datasets.Comment: ICML 201

Psychometric analysis of the UK Health and Safety Executive's Management Standards work-related stress Indicator Tool

In the UK the Health & Safety Executive (HSE) Management Standards (MS) work-related stress Indicator Tool is the standard measure that organizations use to assess workplace stress. However, no in-depth analysis has been performed to test the psychometric properties of this scale. The current study is the first to examine the factor structure of the HSE MS Indicator Tool using organizational-level data. Data collected from 39 organizations (N = 26,382) was used to perform a first-order Confirmatory Factor Analysis (CFA) on the original 35-item seven-factor measurement scale. The results showed an acceptable fit to the data for the instrument. A second-order CFA was also performed to test if the Indicator Tool contains a higher order uni-dimensional measure of work-related stress. These findings also revealed an acceptable fit to the data, suggesting that it may be possible to derive a single measure of work-related stress

A critical analysis on the efficiency of property development approval processes in the City of Cape Town

The Western Cape Government Economic War Room has identified that land-use management in the City of Cape Town is inefficient. Coupled with the fact that there is a housing crisis within the City of Cape Town, it is imperative that such inefficiencies are addressed with urgency. Current development regulations in the City of Cape Town are said to be hindering the involvement of the private sector in the property development space and adding unnecessary delays to the property development sector in general. This paper will argue that a reason for this can be attributed to convoluted legislation linked to property development approval processes, that is being too rigidly interpreted and not administered efficiently. There is therefore a need to understand how the overall development application system is run, especially in relation to the land use and building plan application processes, to assist in identifying the inefficiencies affecting the property development space as a whole. This will allow pragmatic solutions to be formulated and expanded on, to better expound how a more efficient development environment can be created. A further important factor in better understanding the property development space, is comprehending the context within which it functions. Namely, the governance systems which affect it, the laws and regulations applicable to it, and the lack of emphasis on saving time throughout the application process. The purpose of this paper is to show where the inefficiencies lie in the land use management and building development management application processes, and why such inefficiencies may be happening. This paper will also discuss and recommend further topics that should be studied in order to resolve the various issues named. The methodology used to achieve the aforementioned was a mixed method of data collection, which encompassed various interviews with experts working within the property and planning development fields, iterative communication with these professionals, and literature reviews. In sum, there is no one answer to the identified issues as there are many interconnected complexities that must be dealt with in order to address the inefficiencies effectively. What is clear however, is that the current implementation of administrative penalties by the City of Cape Town are causing major capacity issues within the Development Management department and Municipal Planning Tribunal, and which ultimately has a ripple effect on the system as a whole

Generating Private Data Surrogates for Vision Related Tasks

International audienceWith the widespread application of deep networks in industry, membership inference attacks, i.e. the ability to discern training data from a model, become more and more problematic for data privacy. Recent work suggests that generative networks may be robust against membership attacks. In this work, we build on this observation, offering a general-purpose solution to the membership privacy problem. As the primary contribution, we demonstrate how to construct surrogate datasets, using images from GAN generators, labelled with a classifier trained on the private dataset. Next, we show this surrogate data can further be used for a variety of downstream tasks (here classification and regression), while being resistant to membership attacks. We study a variety of different GANs proposed in the literature, concluding that higher quality GANs result in better surrogate data with respect to the task at hand

On the Theoretical Equivalence of Several Trade-Off Curves Assessing Statistical Proximity

The recent advent of powerful generative models has triggered the renewed development of quantitative measures to assess the proximity of two probability distributions. As the scalar Frechet inception distance remains popular, several methods have explored computing entire curves, which reveal the trade-off between the fidelity and variability of the first distribution with respect to the second one. Several of such variants have been proposed independently and while intuitively similar, their relationship has not yet been made explicit. In an effort to make the emerging picture of generative evaluation more clear, we propose a unification of four curves known respectively as: the precision-recall (PR) curve, the Lorenz curve, the receiver operating characteristic (ROC) curve and a special case of R\'enyi divergence frontiers. In addition, we discuss possible links between PR / Lorenz curves with the derivation of domain adaptation bounds.Comment: 10 pages, 3 figure

Detecting Overfitting of Deep Generative Networks via Latent Recovery

State of the art deep generative networks are capable of producing images with such incredible realism that they can be suspected of memorizing training images. It is why it is not uncommon to include visualizations of training set nearest neighbors, to suggest generated images are not simply memorized. We demonstrate this is not sufficient and motivates the need to study memorization/overfitting of deep generators with more scrutiny. This paper addresses this question by i) showing how simple losses are highly effective at reconstructing images for deep generators ii) analyzing the statistics of reconstruction errors when reconstructing training and validation images, which is the standard way to analyze overfitting in machine learning. Using this methodology, this paper shows that overfitting is not detectable in the pure GAN models proposed in the literature, in contrast with those using hybrid adversarial losses, which are amongst the most widely applied generative methods. The paper also shows that standard GAN evaluation metrics fail to capture memorization for some deep generators. Finally, the paper also shows how off-the-shelf GAN generators can be successfully applied to face inpainting and face super-resolution using the proposed reconstruction method, without hybrid adversarial losses

Functional identification and characterizationof the diuretic hormone 31 (DH31) signaling system in the Green shore crab, Carcinus maenas

<p>The functional characterization of crustacean neuropeptides and their cognate receptors has not kept pace with the recent advances in sequence determination and, therefore, our understanding of the physiological roles of neuropeptides in this important arthropod sub-phylum is rather limited. We identified a candidate receptor-ligand pairing for diuretic hormone 31 (DH31) in a neural transcriptome of the crab, Carcinus maenas. In insects, DH31 plays species -specific but central roles in many facets of physiology, including fluid secretion, myoactivity, and gut peristalsis but little is known concerning its functions in crustaceans. The C. maenas DH31 transcript codes for a 147 amino acid prepropeptide, and a single receptor transcript translates to a secretin-like (Class B1) G protein-coupled receptor (GPCR). We used an in vitro aequorin luminescence Ca²⁺ mobilization assay to demonstrate that this candidate DH31R is activated byCarcinus and insect DH31s in a dose-dependent manner (EC₅₀ 15–30 nM). Whole mount immunohistochemical and in situ hybridization localization revealed extensive DH31 expressing neurons throughout the central nervous system, most notably in the abdominal ganglion where large, unpaired cells give rise to medial nerves, which terminate in extensive DH31 immunopositive dendritic fields intimately associated with oesophageal musculature. This system constitutes a large and hitherto undescribed neurohemal area adjacent to key muscle groups associated with the gastric system. DH31 expressing neurons were also seen in the cardiac, commissural, oesophageal, and stomatogastric ganglia and intense labeling was seen in dendrites innervating fore- and hindgut musculature but not with limb muscles. These labeling patterns, together with measurement of DH31R mRNA in the heart and hindgut, prompted us test the effects of DH31 on semi-isolated heart preparations. Cardiac superfusion with peptide evoked increased heart rates (10–100 nM). The neuroanatomical distribution of DH31 and its receptor transcripts, particularly that associated with gastric and cardiac musculature, coupled with the cardio- acceleratory effects of the peptide implicate this peptide in key myoactive roles, likely related to rhythmic coordination.</p

A bioinformatics toolkit: in silico tools and online resources for investigating genetic variation

With the advent of large-scale next-generation sequencing initiatives, there is an increasing importance to interpret and understand the potential phenotypic influence of identified genetic variation and its significance in the human genome. Bioinformatics analyses can provide useful information to assist with variant interpretation. This review provides an overview of tools/resources currently available, and how they can help predict the impact of genetic variation at the deoxyribonucleic acid, ribonucleic acid, and protein level

Early Career ESOL Teachers’ Practical Knowledge of Teaching Speaking

This thesis presents the findings of qualitative multiple-case study research investigating ESOL teachers’ practical knowledge of teaching speaking. Although there has been increased recognition of the value of practical knowledge research in recent years, such research remains extremely limited and the practical knowledge and teachers in an ESOL context and in the curricular domain of teaching speaking skills were previously unexplored areas. The four research participants were all early career ESOL teachers in the United Kingdom. Classroom observation data and interview data were generated at multiple points over the course of an academic year. This methodological approach introduced a longitudinal dimension to the research enabling any possible practical knowledge growth to be investigated. The research identified the largely contemporary nature of the ESOL teachers’ practices in teaching speaking. However, the teachers’ practical knowledge was identified as being atheoretical: teachers did not refer to public theory in the explanations of their practices. Instead, the findings suggest that teachers may experience a process of socialisation (both institutional and sectorial) through which many practices are adopted without a theoretical basis. A significant degree of commonality was identified amongst the teachers’ practical knowledge. Individual differences appeared to be significant, however, and were identifiable both in teachers’ practices and the beliefs underlying them. Teachers’ exercising of significant agency in their practices meant that these differences were evident despite certain sectorial pressure on teachers, particularly through exam washback. There was very limited evidence of growth in the teachers’ practical knowledge of teaching speaking. The research indicated a number of factors which appeared to inhibit such growth. The study discusses the implications of these findings for ESOL teacher development programmes. Recommendations for teacher development programmes include constructivist approaches to teacher engagement with public theory and institutional mechanisms for a sharing of practices amongst teachers