After statistics reform

In the longer term null hypothesis significance testing (NHST) will disappear because p- values are not informative and not replicable. Should we continue to teach in the future the procedures of then abolished routines (i.e., NHST)? Three arguments are discussed for not teaching NHST in introductory courses in inferential statistics: NHST is too difficult for introductory courses; dichotomous thinking is a cognitive obstacle for interpretation; students can find relevant information in research reports without knowing NHST

Theorie toetsen in kwalitatief onderzoek

Inleiding Kwalitatieve onderzoekers betreuren het over het algemeen dat kwalitatief onderzoek doorgaans een exploratieve rol krijgt toebedeeld. In dit essay betoog ik dat zij zelf deels schuldig zijn aan deze roltoedeling, omdat zij geen overtuigende strategie ontwikkeld hebben om theorie te toetsen met kwalitatieve methoden. Ik wil laten zien dat en hoe de gangbare methodologische noties over het ontwikkelen van een generaliseerbare theorie in kwalitatief onderzoek tekortschieten. Ik zal ook proberen een alternatief daarvoor te formuleren

The Theory-Testing Questionnaire Business Survey Has Passed Its Expiry Date

Response rates in business surveys are low, which implies a huge risk of selection bias. Usually no attempt is made to assess the extent of selection bias and published survey results might, therefore, not be a correct reflection of actual population characteristics. In this paper, it is argued that response rates cannot be improved to a sufficient degree and that assessment of selection bias is difficult in practice. It is concluded that academic questionnaire surveys of businesses should be abandoned and that an alternative way of testing theory in populations must be found

Replication

Replication is conducting a study in another case (or population) in order to assess whether a research finding from previous studies can be confirmed. The aim of replication is to assess the generalizability of a theoretical claim and the “research finding” that is (or is not) confirmed in the replication study is a claim that is derived from the previous studies. Sometimes the term replication is also used for conducting a study in the same case (or population) in order to assess whether the finding from the previous study can be reproduced. To avoid confusion, the latter method of assessing the reliability of a study might better be called duplication. Duplication will not be discussed here

Theory-Testing With Cases

Theory-testing with cases is ascertaining whether the empirical evidence in a case or in a sample of cases either supports or does not support the theory. There are two methodologies for theory-testing with cases, (a) testing in a single case (‘theory-testing single case study’), and (b) testing in a sample of cases (‘theory-testing sample case study’). The functional form of the proposition that is tested determines which of these two methodologies should be used

Pattern matching

Pattern matching is comparing two patterns in order to determine whether they match (i.e., that they are the same) or do not match (i.e., that they differ). Pattern matching is the core procedure of theory-testing with cases. Testing consists of matching an “observed pattern” (a pattern of measured values) with an “expected pattern” (a hypothesis), and deciding whether these patterns match (resulting in a confirmation of the hypothesis) or do not match (resulting in a disconfirmation). Essential to pattern matching (as opposed to pattern recognition, which is a procedure by which theory is built) is that the expected pattern is precisely specified before the matching takes place

Theory-Building With Cases

Theory-building with cases is (a) formulating new propositions that emerge from the empirical evidence in a sample of cases and (b) testing them in the same sample. The main difference with most other forms of generating new propositions (such as analyzing the theoretical literature, brainstorming, etc.) is its empirical character. The main difference with other forms of discovering new propositions in empirical evidence (such as in ‘exploratory’ research) is that only those theoretical formulations are accepted as a result of the theory-building study that are confirmed in a test in the sample from which the proposition was built. It is possible that a proposition about a relationship between two variables emerges from an exploratory single case study (e.g., when both variables have extreme values in that case), but it is not possible to test that new proposition in the same study because this would require a comparison in a sample of cases. The term theory-building study (as distinct from an exploratory study) is used here only for studies in which a proper test of the new proposition has been conducted

Not Willing, Not Able: Causes of Measurement Error in Business Surveys

National statistical institutes must collect accurate data from businesses in a timely and cost-effective way and without causing too much response burden. An adequate design of the information request is critical in achieving this goal. This paper describes the lessons we have learned about the design of business survey questionnaires from a thorough evaluation of the questionnaires of a typical business survey for official statistics, the Structural Business Survey. The paper presents a framework for understanding factors that contribute to missing and inaccurate data and draws a number of conclusions regarding how the design of business surveys can be improved to take these factors into account

The Productivity of the Three-Step Test-Interview (TSTI) Compared to an Expert Review of a Self-administered Questionnaire on Alcohol Consumption

The three-step test interview (TSTI) is a recently developed observation-based procedure for the identification of response problems in self-administered survey questionnaires. The TSTI was applied in field test interviews to a quantity-frequency-variability questionnaire on alcohol consumption. For an assessment of its productivity the results are compared to a previously performed expert review. Most response problems that were identified in the expert review were confirmed in the field test interviews. Additionally, the TSTI identified many unexpected problems, mostly stemming from unanticipated “deviant” drinking patterns and from local normative connotations attached to drinking alcohol. From these findings we conclude that the TSTI is a powerful test tool with a high ecological validity

Formulations in first encounters

The paper describes and compares the use and function of the formulation--decision pair in three types of diagnostic interviewing. The investigatory type of interviewing, which typically occurs in the medical interview, is characterized by the absence of formulations. In the exploratory type of interviewing, which typically occurs in the psychiatric interview, the interviewer uses the formulation-decision pair for exploring the patient's experiences. He does not, however, formulate his professional assessment in the interview. In the collaborative type, which typically is found in psychotherapy, the formulation-decision pair is the main interactional device used by the interviewer for translating the patient's troubles into a professional problem definition. It is suggested that the patient's protoprofessionalization, which is typical for psychotherapy, already begins in the initial interview and that it is an effect of the interviewer's specific use of the formulation-decision device in the interview