Probabilistic performance estimators for computational chemistry methods: Systematic Improvement Probability and Ranking Probability Matrix. I. Theory

The comparison of benchmark error sets is an essential tool for the evaluation of theories in computational chemistry. The standard ranking of methods by their Mean Unsigned Error is unsatisfactory for several reasons linked to the non-normality of the error distributions and the presence of underlying trends. Complementary statistics have recently been proposed to palliate such deficiencies, such as quantiles of the absolute errors distribution or the mean prediction uncertainty. We introduce here a new score, the systematic improvement probability (SIP), based on the direct system-wise comparison of absolute errors. Independently of the chosen scoring rule, the uncertainty of the statistics due to the incompleteness of the benchmark data sets is also generally overlooked. However, this uncertainty is essential to appreciate the robustness of rankings. In the present article, we develop two indicators based on robust statistics to address this problem: P_{inv}, the inversion probability between two values of a statistic, and \mathbf{P}_{r}, the ranking probability matrix. We demonstrate also the essential contribution of the correlations between error sets in these scores comparisons

Probabilistic performance estimators for computational chemistry methods: Systematic Improvement Probability and Ranking Probability Matrix. II. Applications

In the first part of this study (Paper I), we introduced the systematic improvement probability (SIP) as a tool to assess the level of improvement on absolute errors to be expected when switching between two computational chemistry methods. We developed also two indicators based on robust statistics to address the uncertainty of ranking in computational chemistry benchmarks: Pinv , the inversion probability between two values of a statistic, and Pr , the ranking probability matrix. In this second part, these indicators are applied to nine data sets extracted from the recent benchmarking literature. We illustrate also how the correlation between the error sets might contain useful information on the benchmark dataset quality, notably when experimental data are used as reference

The impact of inter-organizational management control systems on performance: a longitudinal case study of a supplier relation in automotive.

This study investigates whether appropriate management control design of supplier relations is associated with better performance. Although management control systems (MCSs) are found to be contingent on situational characteristics, it remains unclear whether this contingency fit contributes to performance. In order to illustrate the existence and refine the dynamics of the fit-performance association, we perform a longitudinal case study of an exemplary automotive manufacturer-supplier relation that was subject to considerable change and severe performance difficulties in the course of time. As proposed, case findings show that if the supplier is incapable of dealing with changed contingencies, a MCS contingency misfit is associated with poor operational performance. However, this misfit is only temporal, as the manufacturer adapts the MCS to fit the changed supplier relation and regain operational performance. In addition, the longitudinal study suggests that trust and basic formal control (control continuously exercised under all circumstances) are complements, while trust substitutes for extra formal control (control set up on top of basic formal control). Finally, the data indicate a timing difference in the substitutive relation: the building up of extra formal control proceeds gradually, while the lowering happens almost immediately.management control; trust; performance; supplier relationships; manufacturing; contingency theory; case research; automotive;

Management control of supplier relationships in manufacturing: a case study in the automotive industry.

This paper studies management control design of supplier relationships in manufacturing, a supply chain phase currently under-explored. Compared to supplier relations during procurement and R&D, which research found to be governed by a combination of formal and informal controls, supplier relations in manufacturing are more formal, so that they could be governed by more formal and less informal controls. To refine the management control system and influencing contingencies, we propose a theoretical framework specifically adapted for the manufacturing stage. This framework is investigated by an in depth case study of the supplier management control of a Volvo Cars production facility. We identify three types of suppliers visualizing the associations in the framework and illustrating the framework’s explicative power in (automotive) manufacturing. Furthermore, the case contradicts that supplier relations in the manufacturing phase are governed by little informal control, because the automaker highly values the role of trust building and social pressure. Most notably, a structured supplier team functions as a clan and establishes informal control among participating suppliers, which strengthens the automaker’s control on dyadic supplier relations.management control; supplier relationships; manufacturing; contingency theory; case research;

Repairing triangle meshes built from scanned point cloud

The Reverse Engineering process consists of a succession of operations that aim at creating a digital representation of a physical model. The reconstructed geometric model is often a triangle mesh built from a point cloud acquired with a scanner. Depending on both the object complexity and the scanning process, some areas of the object outer surface may never be accessible, thus inducing some deficiencies in the point cloud and, as a consequence, some holes in the resulting mesh. This is simply not acceptable in an integrated design process where the geometric models are often shared between the various applications (e.g. design, simulation, manufacturing). In this paper, we propose a complete toolbox to fill in these undesirable holes. The hole contour is first cleaned to remove badly-shaped triangles that are due to the scanner noise. A topological grid is then inserted and deformed to satisfy blending conditions with the surrounding mesh. In our approach, the shape of the inserted mesh results from the minimization of a quadratic function based on a linear mechanical model that is used to approximate the curvature variation between the inner and surrounding meshes. Additional geometric constraints can also be specified to further shape the inserted mesh. The proposed approach is illustrated with some examples coming from our prototype software

Probabilistic performance estimators for computational chemistry methods: the empirical cumulative distribution function of absolute errors

Benchmarking studies in computational chemistry use reference datasets to assess the accuracy of a method through error statistics. The commonly used error statistics, such as the mean signed and mean unsigned errors, do not inform end-users on the expected amplitude of prediction errors attached to these methods. We show that, the distributions of model errors being neither normal nor zero-centered, these error statistics cannot be used to infer prediction error probabilities. To overcome this limitation, we advocate for the use of more informative statistics, based on the empirical cumulative distribution function of unsigned errors, namely (1) the probability for a new calculation to have an absolute error below a chosen threshold, and (2) the maximal amplitude of errors one can expect with a chosen high confidence level. Those statistics are also shown to be well suited for benchmarking and ranking studies. Moreover, the standard error on all benchmarking statistics depends on the size of the reference dataset. Systematic publication of these standard errors would be very helpful to assess the statistical reliability of benchmarking conclusions.Comment: Supplementary material: https://github.com/ppernot/ECDF

A survey to evaluate how non designers perceive aesthetic properties of styling features

World-wide market competition and the need to create products that better satisfy the market expectations require a more comprehensive involvement of the customer in the product definition loop. Therefore, it is crucial to provide customers with very easy-to-use shape definition and modification tools, allowing them to verify and evaluate possible shape alternatives without requiring specific knowledge on geometric modeling. A set of aesthetic properties guiding the shape characterization and appraisal have been identified together with measures for their evaluation and shape modeling methods for their direct modification. Since these properties have been indicated by stylists, no guarantee exists that they are usable in a context directly involving customers in the product definition loop. To verify the extent to which the terms indicating the properties, their meaning and their measures are significant and understandable by non-expert designer people we carried out a survey. This paper describes the methodology adopted and the outcomes of this survey