Compositionality and the Explanation of Cognitive Processes

Connectionist approaches to the modeling of cognitive processes have often been attacked on the grounds that they do not employ compositionally structured representations (e.g.,Fodor & Pylyshyn 1988). But what exactly is compositional structure, and how does such structure contribute to cognitive processing? This paper clarifies these questions by distinguishing two different styles of compositionality, one characteristic of connectionist modeling and the other essential to mainstream symbolic or "Classical" approaches. Given this distinction,it is clear that connectionist models can employ compositionally structured representations while remaining, both conceptually and in practice, quite distinct from the Classical approach;moreover, it can be shown that certain central aspects of cognition, such as its systematicity,are at least in principle amenable to Connectionist explanation

Classicalism and Cognitive Architecture

This paper challenges the widely accepted claim that "classical" cognitive architectures can explain the systematicity of cognition (Fodor & Pylyshyn, 1988). There are plausible ways of rendering more precise the systematicity hypothesis (as standardly formulated) in which it is entailed by classical architectures, and other plausible ways in which it is not. Therefore, it is not a determinate issue whether systematicity is entailed, and hence explained, by classical architectures. The general argument is illustrated in a particular domain, the systematicity of deductive inference. In the case of the capacity to carry out the inference modus tollens, the systematicity hypothesis can be made precise in two ways, one entailed by classical architectures, another which is not. Further, the latter, but not the former, accurately describes the actual empirical phenomenon. Put another way, the clumps that these deductive inference capacities come in are not the clumps that are entailed by classical architectures. Therefore, in this area at least, systematicity considerations count against the classical conception of cognitive architecture

Measuring quality of general reasoning

Machine learning models that automatically assess reasoning quality are trained on human-annotated written products. These “gold-standard” corpora are typically created by prompting annotators to choose, using a forced choice design, which of two products presented side by side is the most convincing, contains the strongest evidence or would be adopted by more people. Despite the increase in popularity of using a forced choice design for assessing quality of reasoning (QoR), no study to date has established the validity and reliability of such a method. In two studies, we simultaneously presented two products of reasoning to participants and asked them to identify which product was ‘better justified’ through a forced choice design. We investigated the criterion validity and inter-rater reliability of the forced choice protocol by assessing the relationship between QoR, measured using the forced choice protocol, and accuracy in objectively answerable problems using naive raters sampled from MTurk (Study 1) and experts (Study 2), respectively. In both studies products that were closer to the correct answer and products generated by larger teams were consistently preferred. Experts were substantially better at picking the reasoning products that corresponded to accurate answers. Perhaps the most surprising finding was just how rapidly raters made judgements regarding reasoning: On average, both novices and experts made reliable decisions in under 15 seconds. We conclude that forced choice is a valid and reliable method of assessing QoR

Validating a forced‑choice method for eliciting quality‑of‑reasoning judgments

In this paper we investigate the criterion validity of forced-choice comparisons of the quality of written arguments with normative solutions. Across two studies, novices and experts assessing quality of reasoning through a forced-choice design were both able to choose arguments supporting more accurate solutions—62.2% (SE = 1%) of the time for novices and 74.4% (SE = 1%) for experts—and arguments produced by larger teams—up to 82% of the time for novices and 85% for experts—with high inter-rater reliability, namely 70.58% (95% CI = 1.18) agreement for novices and 80.98% (95% CI = 2.26) for experts. We also explored two methods for increasing efficiency. We found that the number of comparative judgments needed could be substantially reduced with little accuracy loss by leveraging transitivity and producing quality-of-reasoning assessments using an AVL tree method. Moreover, a regression model trained to predict scores based on automatically derived linguistic features of participants’ judgments achieved a high correlation with the objective accuracy scores of the arguments in our dataset. Despite the inherent subjectivity involved in evaluating differing quality of reasoning, the forced-choice paradigm allows even novice raters to perform beyond chance and can provide a valid, reliable, and efficient method for producing quality-of-reasoning assessments at scale