Behavioral observation versus behavioral expectation rating scales: Development and psychometric properties

Rating scales to assess the level of proficiency of human performance have come in a wide variety of formats. The objectives of the scales, types of questions asked, and procedures followed vary widely. Prior to 1960, most rating scales utilized what is known as the graphic format. According to Guion (1965) the rating scale was relied on in 81% of the validation studies using rating scales. While this has declined somewhat (Landy § Trumbo, 1976) performance ratings still play a major part in validation. This study will compare behavioral observation and behavioral expectation scales in terms of psychometric properties. Central to these scales is the question whether either of the scale development procedures is superior

Using Archival Data for I-O Research: Advantages, Pitfalls, Sources, and Examples

Two particular sets of experiences sparked our interest in writing this TIP article. The first was our increasing difficulty getting access to “new” organization- based samples. Depending on the topic and commitment involved, many organizations appear too leery and/or too strapped these days to allow for primary data collection. In addition, we have all experienced the disappointment of spending numerous hours on research proposals and meetings with organizational personnel, only to have the “plug pulled” at the last minute on a promising line of data collection. Conversely, we have also had experience with researchers in organizations who are willing and interested in partnering to analyze existing company data. A second experience that sparked our interest was supervising graduate student theses and dissertations. Students likely have even more difficulty than faculty in gaining access to organization-based samples. As a result, they often end up collecting survey data on “working students” or other campusbased convenience samples. Although we realize that “working students” may often be appropriate subjects, depending on the research questions being asked, it has been our experience that students often resort to this strategy even when it may not be appropriate, once they find they can’t obtain access to organization-based samples. Given these experiences, we thought a short TIP article outlining some of the key issues of using archival data for I-O research would be of interest to many TIP readers. We by no means foresee (or propose) the use of archival data sets becoming the principal “data collection strategy” within I-O psychology. Rather, we see this as an underutilized tool to be added to current and future I-O psychologists’ methodological toolbox. Given our extensive experiences working with a variety of sources of archival data, we realize there are numerous issues about which someone new to the area needs to be aware. Given the necessary brevity of a TIP article, we refer readers to key references cited throughout the rest of the paper for a detailed discussion of the issues raised below

The North American tree-ring fire-scar network

Fire regimes in North American forests are diverse and modern fire records are often too short to capture important patterns, trends, feedbacks, and drivers of variability. Tree-ring fire scars provide valuable perspectives on fire regimes, including centuries-long records of fire year, season, frequency, severity, and size. Here, we introduce the newly compiled North American tree-ring fire-scar network (NAFSN), which contains 2562 sites, >37,000 fire-scarred trees, and covers large parts of North America. We investigate the NAFSN in terms of geography, sample depth, vegetation, topography, climate, and human land use. Fire scars are found in most ecoregions, from boreal forests in northern Alaska and Canada to subtropical forests in southern Florida and Mexico. The network includes 91 tree species, but is dominated by gymnosperms in the genus Pinus. Fire scars are found from sea level to >4000-m elevation and across a range of topographic settings that vary by ecoregion. Multiple regions are densely sampled (e.g., >1000 fire-scarred trees), enabling new spatial analyses such as reconstructions of area burned. To demonstrate the potential of the network, we compared the climate space of the NAFSN to those of modern fires and forests; the NAFSN spans a climate space largely representative of the forested areas in North America, with notable gaps in warmer tropical climates. Modern fires are burning in similar climate spaces as historical fires, but disproportionately in warmer regions compared to the historical record, possibly related to under-sampling of warm subtropical forests or supporting observations of changing fire regimes. The historical influence of Indigenous and non-Indigenous human land use on fire regimes varies in space and time. A 20th century fire deficit associated with human activities is evident in many regions, yet fire regimes characterized by frequent surface fires are still active in some areas (e.g., Mexico and the southeastern United States). These analyses provide a foundation and framework for future studies using the hundreds of thousands of annually- to sub-annually-resolved tree-ring records of fire spanning centuries, which will further advance our understanding of the interactions among fire, climate, topography, vegetation, and humans across North America

GA4GH: International policies and standards for data sharing across genomic research and healthcare.

The Global Alliance for Genomics and Health (GA4GH) aims to accelerate biomedical advances by enabling the responsible sharing of clinical and genomic data through both harmonized data aggregation and federated approaches. The decreasing cost of genomic sequencing (along with other genome-wide molecular assays) and increasing evidence of its clinical utility will soon drive the generation of sequence data from tens of millions of humans, with increasing levels of diversity. In this perspective, we present the GA4GH strategies for addressing the major challenges of this data revolution. We describe the GA4GH organization, which is fueled by the development efforts of eight Work Streams and informed by the needs of 24 Driver Projects and other key stakeholders. We present the GA4GH suite of secure, interoperable technical standards and policy frameworks and review the current status of standards, their relevance to key domains of research and clinical care, and future plans of GA4GH. Broad international participation in building, adopting, and deploying GA4GH standards and frameworks will catalyze an unprecedented effort in data sharing that will be critical to advancing genomic medicine and ensuring that all populations can access its benefits

The James Webb Space Telescope Mission

Twenty-six years ago a small committee report, building on earlier studies, expounded a compelling and poetic vision for the future of astronomy, calling for an infrared-optimized space telescope with an aperture of at least $4m$. With the support of their governments in the US, Europe, and Canada, 20,000 people realized that vision as the $6.5m$ James Webb Space Telescope. A generation of astronomers will celebrate their accomplishments for the life of the mission, potentially as long as 20 years, and beyond. This report and the scientific discoveries that follow are extended thank-you notes to the 20,000 team members. The telescope is working perfectly, with much better image quality than expected. In this and accompanying papers, we give a brief history, describe the observatory, outline its objectives and current observing program, and discuss the inventions and people who made it possible. We cite detailed reports on the design and the measured performance on orbit.Comment: Accepted by PASP for the special issue on The James Webb Space Telescope Overview, 29 pages, 4 figure

A TEST OF THE ASSUMED SUPERIORITY OF THE EXPERT RATER

The ratings of expert raters have been used as benchmark or true-score criteria in previous research regarding the performance appraisal process. Such experts typically have a large amount of training and/or experience with regard to performance appraisal, and their ratings are assumed to be superior to ratings produced by less experienced raters. The superiority of expert raters has not been experimentally tested within the industrial psychology literature, although experienced and highly trained clinical psychologists have not demonstrated clear superiority in observation and judgmental skills within the clinical literature. Previous research on expert raters has confounded expertise of the raters with the amount of exposure to rating stimuli such as videotapes and rating scales. The present study investigated the effects of expertise and review of rating materials in a factorial design. As hypothesized, rater expertise was less important than was level of exposure to rating stimuli. Results are discussed in terms of the implications of these findings on previous as well as on future research utilizing expert raters

A Comparison of Scoring Strategies for Operational Situational Judgment Tests

The use of situational judgment tests (SJTs) has increased recently. SJTs offer many advantages compared to other assessments, including validity, flexibility in administration, objective scoring, and reduced mean group differences. While use has increased (Roth, Bobko, & Buster, 2013), there is little consensus regarding SJT scoring (Bergman et al., 2006). This study used data from operational SJTs to evaluate and compare scoring strategies in terms of convergence, reliability, mean group differences, and candidate rank order. Results demonstrate that different scoring strategies can produce wildly different rank orders and can contribute to differences in reliability values and mean group test scores