Boosting intelligence analysts’ judgment accuracy: what works, what fails?

A routine part of intelligence analysis is judging the probability of alternative hypotheses given available evidence. Intelligence organizations advise analysts to use intelligence-tradecraft methods such as Analysis of Competing Hypotheses (ACH) to improve judgment, but such methods have not been rigorously tested. We compared the evidence evaluation and judgment accuracy of a group of intelligence analysts who were recently trained in ACH and then used it on a probability judgment task to another group of analysts from the same cohort that were neither trained in ACH nor asked to use any specific method. Although the ACH group assessed information usefulness better than the control group, the control group was a little more accurate (and coherent) than the ACH group. Both groups, however, exhibited suboptimal judgment and were susceptible to unpacking effects. Although ACH failed to improve accuracy, we found that recalibration and aggregation methods substantially improved accuracy. Specifically, mean absolute error (MAE) in analysts’ probability judgments decreased by 61% after first coherentizing their judgments (a process that ensures judgments respect the unitarity axiom) and then aggregating their judgments. The findings cast doubt on the efficacy of ACH, and show the promise of statistical methods for boosting judgment quality in intelligence and other organizations that routinely produce expert judgments

Arithmetic computation with probability words and numbers

Probability information is regularly communicated to experts who must fuse multiple estimates to support decision-making. Such information is often communicated verbally (e.g., “likely”) rather than with precise numeric (point) values (e.g., “.75”), yet people are not taught to perform arithmetic on verbal probabilities. We hypothesized that the accuracy and logical coherence of averaging and multiplying probabilities will be poorer when individuals receive probability information in verbal rather than numerical point format. In four experiments (N = 213, 201, 26, and 343, respectively), we manipulated probability communication format between-subjects. Participants averaged and multiplied sets of four probabilities. Across experiments, arithmetic accuracy and coherence was significantly better with point than with verbal probabilities. These findings generalized between expert (intelligence analysts) and non-expert samples and when controlling for calculator use. Experiment 4 revealed an important qualification: whereas accuracy and coherence were better among participants presented with point probabilities than with verbal probabilities, imprecise numeric probability ranges (e.g., “.70 to .80”) afforded no computational advantage over verbal probabilities. Experiment 4 also revealed that the advantage of the point over the verbal format is partially mediated by strategy use. Participants presented with point estimates are more likely to use mental computation than guesswork, and mental computation was found to be associated with better accuracy. Our findings suggest that where computation is important, probability information should be communicated to end users with precise numeric probabilities

Cultural and ethical effects on managerial decisions : examined in a throughput model.

Financial and cost accounting information is processed by decision-makers guided by their particular need to support decisions. Recent technological advances impacting on information as well as organizations such as the European Community mandating financial reporting requirements for many countries is rapidly changing the landscape for decision making using accounting information. Hence, the importance of individuals'' decision making is more important than it was previously. These decisions are also influenced by individuals'' ethical beliefs. The Throughput Modeling approach to cultural and ethical concerns provides a way of dealing with accounting information processed through various pathways by decision-makers. This modeling approach captures different philosophical perspectives from which to understand what is involved in "thinking scientifically." In the Throughput Modeling approach, pathways highlight the importance of how different philosophical perspectives may be used by individuals in arriving at a decision. This paper highlights key concepts involved in rethinking the basis of moral decision making in terms of an underlying process, rather than focusing on the application of principles or the development of a virtuous character. Examples are provided from both English and Spanish settings to help emphasize the importance of modeling ethical decision making globally.Decision making; Ethical behavior; Judgment and choice;

Image and Reputational Impact on Managers' Assessment of Auditing Activities

The efficient use of knowledge assets allows organizations to compete more effectively in the marketplace. Some of these knowledge assets are embedded in an organization socialization and internalization processes. Image and reputation may better leverage knowledge by strengthening its core capabilities of knowledge assets. Image and reputation is viewed as a collective 'perception' pertaining to a person, group, or organization's attributes. We suggest that by combining the use of reputational data and structural equation modeling of narrative analysis, we can build a better process theory along with better explanations in general. Throughput modeling techniques are used in this research to capitalize on how line managers' narratives are captured in framing their international environment, the information used, as well as deciding whether audits are assisting them in promoting increase profits or decrease expenses. This model is useful in highlighting the elements that support a view of managers being proactive. This analysis captures forward looking throughput modeling techniques whereby in-house managerial procedures can help aid and assist line managers performing their primary functions. Findings indicated that managers' perception of image/reputation influence what they viewed as important knowledge assets relating to productivity and profitability. --

MATHEMATICAL MODEL FOR MANAGEMENT OF THE MUNICIPAL AUTHORITIES

In the presented paper to solve the problem was used the "Analytic Hierarchy Process" method developed by Tomas L. Saaty. AHP is one of the most popular analytical techniques for complex decision-making problems. In this method a decision-making problem decomposes into a system of hierarchies of objectives, attributes (or criteria), and alternatives. Then to obtain optimum solution uses judgements of experts with a special scale for measuring non-quantitative and method of establishing priorities.In the presented paper to solve the problem was used the "Analytic Hierarchy Process" method developed by Tomas L. Saaty. AHP is one of the most popular analytical techniques for complex decision-making problems. In this method a decision-making problem decomposes into a system of hierarchies of objectives, attributes (or criteria), and alternatives. Then to obtain optimum solution uses judgements of experts with a special scale for measuring non-quantitative and method of establishing priorities

Aquatic Invasive Species Change Ecosystem Services from the World\u27s Largest Wild Sockeye Salmon Fisheries in Alaska

This study combines a multi-method approach to structured expert judgment with market valuation to forecast fisheries damages from introduced invasive species. The method is applied to a case study of Alaska’s first submersed aquatic invasive plant, Elodea spp., threatening Alaska’s salmon fisheries. Assuming that Elodea spp. remains unmanaged, estimated mean damages to commercial sockeye fisheries aggregated across Alaska amount to a potential $159 million annually with a 5$577 million annually ($2015 USD). The associated mean loss of natural capital amounts to$5.1 billion cumulatively over the next 100 years reaching $400 million after 10 years. Results from the expert elicitation indicate that there is a 35% chance of positive net benefits associated with the believed positive effects of Elodea spp. on sockeye salmon (Oncorhynchus nerka). Despite the potential for positive net gains, the magnitude of the most probable damage estimate may justify substantial investment in keeping productive freshwater systems free of aquatic invasive species. The damage estimate for Alaska is significantly larger than similar estimates in the Great Lakes where ecosystems are already impaired by multiple aquatic invasive species, underscoring the value of keeping functioning ecosystems with global market value productive. This study is the first to estimate ecosystem service loss associated with introduction of an aquatic invasive species to freshwater habitat that supports the world’s most valuable wild sockeye salmon fisheries. Important policy implications related to natural resource management and efficient allocation of scarce resources are discusse

Aquatic Invasive Species Change Ecosystem Services from the World�s Largest Wild Sockeye Salmon Fisheries in Alaska

This study combines a multi-method approach to structured expert judgment with market valuation to forecast fisheries damages from introduced invasive species. The method is applied to a case study of Alaska�s first submersed aquatic invasive plant, Elodea spp., threatening Alaska�s salmon fisheries. Assuming that Elodea spp. remains unmanaged, estimated mean damages to commercial sockeye fisheries aggregated across Alaska amount to a potential $159 million annually with a 5$577 million annually ($2015 USD). The associated mean loss of natural capital amounts to$5.1 billion cumulatively over the next 100 years reaching $400 million after 10 years. Results from the expert elicitation indicate that there is a 35% chance of positive net benefits associated with the believed positive effects of Elodea spp. on sockeye salmon (Oncorhynchus nerka). Despite the potential for positive net gains, the magnitude of the most probable damage estimate may justify substantial investment in keeping productive freshwater systems free of aquatic invasive species. The damage estimate for Alaska is significantly larger than similar estimates in the Great Lakes where ecosystems are already impaired by multiple aquatic invasive species, underscoring the value of keeping functioning ecosystems with global market value productive. This study is the first to estimate ecosystem service loss associated with introduction of an aquatic invasive species to freshwater habitat that supports the world�s most valuable wild sockeye salmon fisheries. Important policy implications related to natural resource management and efficient allocation of scarce resources are discusse

Using Experts' Beliefs to Inform Public Policy: Capturing and Using the Views of Many

Cost-effectiveness decision modelling (CEDM) is widely used to inform healthcare resource allocation, however there is often a paucity of data to quantify the level of uncertainty around model parameters. Expert elicitation has been proposed as a method for quantifying uncertainty when other sources of evidence are not available. Elicitation refers to formal processes for quantifying experts’ beliefs about uncertain quantities, typically as probability distributions. It is generally conducted with multiple experts to minimise bias and ensure representation of experts with different perspectives. In CEDM, priors are most commonly elicited from individual experts then pooled mathematically into an aggregate prior that is subsequently used in the model. When pooling priors mathematically, the investigator must decide whether to weight all experts equally or assume that some experts in the sample should be given ‘more say’. The choice of method for deriving weights for experts’ priors can affect the resulting estimates of uncertainty, yet it is not clear which method is optimal. This thesis develops an understanding of the methods for deriving weights in opinion pooling. A literature review is first conducted to identify the existing methods for deriving weights. Differences between the identified methods are then analysed and discussed in terms of how they affect the role of each method in elicitation. The developed principles are then applied in a case study, where experts’ priors on the effectiveness of a health intervention are elicited, and used to characterise parametric uncertainty in a CEDM. The findings are used to analyse and compare different methods for weighting priors, and to observe the consequences of using different methods in the decision model. The findings improve the understanding of how different weighting methods capture experts’ ‘contributions’ while the choice of methods for deriving weights is found to influence the decision generated by the model

Assessing the communication quality of CSR reports. A case study on four Spanish food companies

This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution license.This article belongs to the Section Economic, Business and Management Aspects of Sustainability.Sustainability reports are tools for disseminating information to stakeholders and the public, serving the organizations in the dual purpose of communicating CSR and being accountable. The production of these reports has recently become more prevalent in the food industry, despite the fact this practice has received heavy criticism on two fronts: The quality of the tool for communication, and the extent of accountability. In addition to these criticisms, organizations must overcome the additional challenge of publishing sustainability reports that successfully meet the demands of a multi-stakeholder audience. In light of the importance of this practice, this paper presents a method to assess the communication and accountability characteristics of Spanish food companies' sustainability reports. This method is based on the method Analytic Network Process (ANP) and adopts a multi-stakeholder approach. This research, therefore, provides a reference model for improving sustainability reports, with the aim of successfully meeting their communication objectives and the demands of all stakeholders.This research has been conducted within the research activities of the Master in Corporate Social Responsibility at the Universitat Politècnica de València (http://www.master-rsc.upv.es/).We acknowledge support by the CSIC Open Access Publication Initiative through its Unit of Information Resources for Research (URICI).Peer Reviewe